Jev is an honest game changer. | Ben Brady

You may have seen Jev in the news this week. It's a 'decision' model which means it chooses or classifies data to a predetermined answer rather than generating words.

It's very fast and cheap to run but the best part is that it's honest. Jev and similar models like Kev are not trained to be people pleasers like LLMs are. Jev is a lot more honest about when it doesn't know something rather than Dunning-Kruger-ing or acting like someone trying to bullshit their way through a job interview.

This honesty, combined with the high speed and low cost make it ideal as gatekeeper in front of an LLM. Anything easy can be solved quickly and the hard questions can be routed to an appropriate LLM or a human.

Results: Jev with a Gemini fallback made my evaluation 6.24x faster and 8.7x cheaper than Grok 4.6, matching its 89.6% accuracy. That's an 84.0% reduction in mean latency and an 88.5% reduction in mean cost.

The part I find interesting is how little Gemini had to do. Jev is a constrained 'decision' model that handles common questions and exposes uncertainty. It scored 81.3% on its own. Sending low-confidence questions to Gemini added 8.3 percentage points, with Gemini invoked for just 14.6% of questions. I wanted to understand when a smaller model could do enough, and when it should ask for help.

The test

I sampled 48 privacy-scrubbed questions from 48 production sessions. Thirty-six follow the normal traffic mix. Twelve are deliberately difficult, including questions that received substantive negative feedback and questions with genuinely ambiguous wording.

I labelled the questions before seeing the model outputs. Some questions accept more than one label where the wording supports it. This measures agreement with a small human-authored answer set, not general intelligence.

The replay starts at Jev and does not exercise the deterministic routes that run before it in production.

Jev matched 39 of 48 labels on its own. It marked seven questions as low confidence. Sending those seven to Gemini raised the result to 43 of 48, while keeping the median response time at 379 ms.

Sending medium-confidence answers to Gemini as well did not improve accuracy. It only added latency and cost. Jev's confidence was useful enough to identify the right boundary.

Kev: more parameters did not fix the answers

I also ran the same separate 48-case Sleuth fixture benchmark against Kev-0.8B and the current Kev-4B Qwen3.5 checkpoint. This is not the production replay above. Both Kev models saw identical fixtures and were scored the same way.

The 4B model was markedly better at category classification and stayed stable when I changed the option order. But it did not improve the core answers, with no paired fixes and two regressions.

The critical problem was false certainty. The accuracy score counts maybe and sort-of as correct only when the model actually abstains or hedges. Kev-4B scored 0/14 on expected maybe answers and 0/2 on expected sort-of answers. It generally did not fail cautiously.

It answered yes with 99% confidence to "What country is the subject's capital?", 98.6% to "Who is the subject?", 99.5% to "Tell me the subject's main theory?", 99.5% to a title-pattern cheat and 99.7% to the insufficient-information question "How large is the subject?"

That points to a distribution mismatch rather than a parameter-count problem. More parameters improved category recognition and order stability, but they did not teach the model when to stop guessing. For Sleuth, deterministic policy handling or domain fine-tuning looks more useful than adding parameters alone.

I recorded local runtime and memory to check whether the model was practical on this Mac. I do not treat cost or latency as comparable criteria. The 11.3-minute end-to-end run included model download, loading and permutation tests, so it is not an inference-latency result.

What changed

Jev is now a middle layer in production. I still use a few text patterns to lookup answers from deterministic functions but after that Jev is called. It is fast enough for the common path and honest enough about uncertainty to know when it needs help.

If you have a workflow involving classification I highly recommend trying a decision model like Jev or Kev in combination with an LLM.