TypeSafe's new decision model held a straight on a board showing three spades. I gave it the opponent's exact cards, a made flush. It said it was ahead and shoved 89.5 into a 22.5 pot, five runs out of five.
There are plenty of good uses for Jev and plenty still to find. There is also going to be a regrettable stretch of this hype cycle, because people are wiring it into decisions without evaluating it. TypeSafe skipped published evals in this release, and I think that will produce a lot of naive deployments.
I have built high-performance classification systems for critical domains, so I am skeptical by default. Poker is a cheap way to check: a solver computes the correct play, money rides on it, and plenty of spots are obvious to any player. Here is the eval.
1. The setup
I solved one flop with TexasSolver: heads-up, 100bb deep, LJ opens and BTN calls, flop Q♠ 9♦ 4♠. Eight minutes, 7.6GB, 0.59% exploitability. That gives the correct strategy for every hand either player can hold, on every turn card, at every decision.
Jev gets a state object describing the table the way a player sees it, and one question: which action should hero take, from the options the solver's tree offers. Pot odds, stack-to-pot ratio, hero's hand rank and outs are computed in Python first, so Jev never does arithmetic.
The exact state and question
{
"game": {
"format": "No-limit Texas Hold'em cash game",
"players_dealt_in": 8,
"blinds_bb": {
"small_blind": 0.5,
"big_blind": 1
},
"units": "All amounts are in big blinds (bb)"
},
"seats": [
{
"position": "UTG",
"starting_stack": 100.0,
"status": "folded preflop"
},
{
"position": "UTG+1",
"starting_stack": 100.0,
"status": "folded preflop"
},
{
"position": "LJ",
"starting_stack": 100.0,
"status": "in hand",
"stack_behind": 89.5,
"is_hero": true
},
{
"position": "HJ",
"starting_stack": 100.0,
"status": "folded preflop"
},
{
"position": "CO",
"starting_stack": 100.0,
"status": "folded preflop"
},
{
"position": "BTN",
"starting_stack": 100.0,
"status": "in hand",
"stack_behind": 89.5
},
{
"position": "SB",
"starting_stack": 100.0,
"status": "folded preflop"
},
{
"position": "BB",
"starting_stack": 100.0,
"status": "folded preflop"
}
],
"hero": {
"position": "LJ",
"hole_cards": [
"King of diamonds",
"10 of diamonds"
]
},
"board": {
"flop": [
"Queen of spades",
"9 of diamonds",
"4 of spades"
],
"turn": "Jack of hearts"
},
"action_history": {
"preflop": [
"UTG folds",
"UTG+1 folds",
"LJ (hero) raises to 2.5",
"HJ folds",
"CO folds",
"BTN calls 2.5",
"SB folds",
"BB folds"
],
"flop": [
"LJ (hero) bets 2",
"BTN raises to 8",
"LJ (hero) calls 6"
],
"turn": []
},
"pot": {
"current_pot": 22.5
},
"hero_hand": {
"made_hand": "Straight, King high",
"draws": [],
"outs_to_straight_or_better": 0,
"chance_to_hit_on_next_card": 0.0
},
"decision": {
"street": "turn",
"hero_to_act": true,
"legal_actions": [
"check",
"all-in 89.5"
],
"effective_stack_behind_before_acting": 89.5,
"facing_bet": 0,
"stack_to_pot_ratio": 3.98
}
}{
"action": {
"type": "choice",
"instructions": "Hero is on the turn and it is hero's turn to act. Which action from `decision.legal_actions` should hero take?",
"criteria": {
"check": "Check",
"all_in": "All-in 89.5"
}
}
}Option labels carry no framing. Five phrasings over 30 spots, including "which action makes hero the most money in the long run" and no labels at all, matched the solver on the same 63% of spots. Value-laden verbs pushed Jev toward bigger bets, so the final wording has none.
2. An easy spot: good
Hero holds J♠T♠ on Q♠ 9♦ 4♠ 2♥ facing a bet of 8 into 10.5. Fifteen outs, needs 30% equity to call, has 33%.
Jev calls at 94%, the solver calls at 96%. 215ms. Across 30 random spots it matched the solver's top action 63% of the time.
3. A basic spot: wrong
Different line, same board. Hero bets the flop, BTN raises, hero calls, and the J♥ arrives. Hero holds K♦T♦, which makes K-Q-J-T-9, the best hand possible here. No flush is out there.
HERO K♦10♦ BOARD Q♠9♦ 4♠J♥
Pot 22.5, both players 89.5 behind, hero first to act. The only two actions here are check or all-in for 89.5, four times the pot. Check-or-shove spots are the easiest kind to reason about: nobody shoves four times the pot with a bad hand, so the answer does not depend on sizing. Other spots in this eval have full bet menus.
This is a trap spot that regular players know: you hold the nuts, so shoving four times the pot gets called by nothing you beat, and folds out every hand that would have paid you. You check and let the other player keep betting.
Jev shoved in sixteen runs out of sixteen.
4. Show it the opponent's cards
Now change one card. The turn is the J♠ instead of the J♥, so three spades are showing and hero's straight loses to any flush. Everything else is identical.
HERO K♦10♦ BOARD Q♠9♦ 4♠J♠
Jev shoves here too, 60% across sixteen runs. So I told it what the opponent had: the literal cards, added to the state as villain_hand, plus a second question, is hero's hand better than the hand in villain_hand?
Three flushes shown, three times it answered that hero was ahead. It shoved in all five rows, whether hero was drawing dead or far ahead. On the J♥ board, where no flush is possible, the same question got all nine comparisons right.
5. What changes the answer
I added facts one at a time and ran each version several times.
The last column is Jev's probability for the action it chose: the mean, then the range across runs. Five runs per row, sixteen for the first row and eleven for the second, which I sampled more than once. Run-to-run movement is a few points; the jumps between rows are 15 to 30.
The first three rows contain everything a player sees at the table. None of it changed the decision. The answer changes when the state names the opponent's hand, and moves further as the state adds both hands together, then that hero is behind, then that hero has zero outs. In the last row the state contains the conclusion.
The field that flips it, verbatim
"villain_hand": {
"hole_cards": ["Ace of spades", "King of spades"],
"hand_rank": "Flush, Ace high"
}With the cards alone, Jev shoves across eleven runs, 0.56 to 0.60. With hand_rank added, check wins every run.
6. Haiku 4.5 with thinking off
The spot from section 3, the one where hero holds the nuts. Same two options, no opponent cards, no hand ranks, the same state Jev shoved 62% of the time.
It states that a shove of four times the pot only gets called by better hands, cites the stack-to-pot ratio of 3.98, and checks.
7. The whole range at one decision
Different spot from the same solve. The flop checks through and the A♥ arrives. LJ raised before the flop, so LJ holds most of the aces. The solver bets 71% of its range here, including hands with nothing.
I asked Jev the same question with every hand class that reaches this spot, 54 of them, one call each. Left is the solver, right is Jev. Green is check, amber and orange are the two bet sizes.
Jev bets when its own cards are good and checks when they are not. The solver bets this board with its whole range, because the ace fits the hands LJ raised with. Confidence is 0.86 on checking bottom pair, which the solver bets 84% of the time, and 0.09 on top pair, where Jev is closest to the solver.
8. Asking it the way the docs prescribe
TypeSafe's docs say not to ask one broad question. Ask narrow judgments, send them in one call, combine them in code. I built that and two other shapes, and scored all of them on the same 150 random spots.
- One question. Which action should hero take.
- Six judgments. Hand rank against his betting range, which range the board favors, whether a free card beats hero, whether hero is behind when called, how often he folds, whether his line looks strong. Code turns the six answers into an action.
- Regret per action. One yes/no per available action: taking this action is a mistake. Code picks the lowest.
- Six binary facts. Best hand now, can he have better, will worse hands call, will better hands fold, can hero improve, is a free card dangerous. Then ordinary poker logic in code.
I also scored three rules with no model in them: check if checking is legal otherwise call, pick at random, and always bet. None of them see the cards.
Every number below is how often a strategy picked the action the solver plays most often in that spot. Perfect play scores 100%. The spots are skewed: the solver checks in 95 of the 150, so the second column drops those and keeps only the 55 where it does something else.
Read the first column as a warning about the sample, not a result: a rule that always checks scores 72% there because checking is usually right. In the second column, where the solver bets, raises, calls or folds, Jev's four shapes score 33% to 44% against 24% to 29% for the no-model rules. So Jev is doing something, and it gets the answer wrong more often than it gets it right.
On the 21 spots facing a bet, the six judgments scored 71% against 62% for the single question. Across all spots the six judgments changed 47 actions: 12 matched the solver where the single question had not, 21 went the other way. Tuning thresholds on half the spots and scoring the other half gave 65%.
The judgments are cached, so rewriting the composition rules costs nothing. The whole sweep was 300 calls and 363,000 input tokens, about a cent and a half.
In all four shapes, Jev checks and folds hands that the solver bets.
One caveat on the decomposed versions: the showcases are not decomposed. They hand the model a diff and ask whether it is secure, or a thread and ask whether to refund. That is the undecomposed use, and it is the use being sold.
Limits of this eval
Everything comes from one solved board. I measured agreement with the solver's top action, not EV loss, which is what costs money. The same request run sixteen times put the shove between 0.59 and 0.65, so differences under six points are noise. Evaluating five cards is mechanical, and TypeSafe's docs say to compute that in code rather than ask the model, which is what section 5 ends up doing.
Takeaway
TypeSafe does not publish standard benchmarks. They built their own workflow evals against expensive reference models, and their launch post lists the caveats, including that those workflows were built in-house. I prefer that to a leaderboard. It also means there is no number telling you whether Jev clears the bar for the decision you are about to hand it.
You have to evaluate every situation you want to use this in, one at a time, against an answer key you trust, and decide what a wrong call costs you there. I aligned my setup as carefully as I could, including splitting the question the way their docs prescribe, and it still scored under a one-line rule.
TexasSolver console build, flop Q♠9♦4♠, LJ opens 2.5 and BTN calls, 100bb, 300 iterations to 0.59% exploitability. Jev calls via POST /v1/systemone, model jev-latest resolving to jev-1.13.0, roughly 1,200 input tokens per call at 200 to 300ms.