Confidence-Based Routing: When to Trust Your Classifier

You have a classifier in production. It labels support tickets, routes messages, detects intent. It gives you a prediction and a confidence score. How do you decide when to trust it?

The obvious answer: set a threshold. If confidence is above 0.8, auto-route. Below that, send to a human

That works until it doesn't

Confidence alone is misleading

Consider two predictions on a 77-class banking intent classifier:

Prediction A has low confidence (0.45) but a clear winner. The classifier is pretty sure it's "transaction_charged_twice" since the next best option is far behind at 0.10. This is probably correct

Prediction B has similar confidence (0.22) but three classes are bunched together. The classifier genuinely doesn't know. This needs a human

A simple confidence threshold treats both the same. The margin between #1 and #2 tells them apart

The margin

The margin is the gap between the top prediction's confidence and the second-best prediction's confidence. It tells you how decisive the classifier is

The Jeffy API now includes margin in every prediction response:

curl -s -X POST https://playground.jeffyclassify.com/v1/predict \
  -H "Content-Type: application/json" \
  -d '{"task":"banking77","text":"I was charged twice for the same transaction"}' \
  | python3 -m json.tool

{
  "label": "transaction_charged_twice",
  "confidence": 0.7234,
  "margin": 0.5891,
  "entropy": 0.2103,
  "probabilities": {
    "transaction_charged_twice": 0.7234,
    "card_payment_not_recognised": 0.1343,
    "extra_charge_on_statement": 0.0412,
    ...
  }
}

Three-tier routing

Combine confidence and margin to build a routing system that knows what it knows

In Python:

from jeffy import Engine

engine = Engine()

def route(text, task="clinc_oos"):
    result = engine.predict(task, text)
    conf = result["confidence"]
    margin = result["margin"]

    if conf > 0.8 and margin > 0.2:
        return "auto", result["label"]
    elif conf > 0.5:
        return "suggest", result["label"]
    else:
        return "escalate", None

The thresholds depend on your use case. A misrouted support ticket is annoying. A misclassified medical alert is dangerous. Tune accordingly

Real examples with clinc_oos

clinc_oos covers 150 intents across 10+ industries (banking, travel, food, smart home, auto) plus an out-of-scope class. 88.5% accuracy on 151 classes. Good for testing routing since it has enough classes to produce genuinely ambiguous cases

"Book a flight" and "what's my balance" get auto-routed. High confidence, high margin. The classifier is sure

"Remind me about the meeting" lands in the suggest tier. The classifier picks "reminder" but "meeting_schedule" and "calendar" are close behind. A human should confirm

"I'm bored" is out-of-scope for this classifier. Low confidence, near-zero margin. This is where a fallback kicks in: pass it to an LLM or decision model that can handle open-ended inputs, not straight to a human queue

The escalation tier is also a feedback signal. If you log what gets escalated, you're building a dataset of exactly the cases your classifier can't handle yet. Retrain on that data and the boundary moves. Logistic regression heads retrain in seconds, so this loop can run continuously

Why not just ask the LLM how confident it is?

LLMs generate confidence as text. They write "0.85" the same way they write any other token: as a plausible continuation of what came before. Asking again can give a different number. Rewording the question shifts the score. The number isn't a measurement, it's a narrative

Logistic regression produces naturally well-calibrated probabilities. The confidence comes from the training data distribution, not from a model narrating how sure it feels. Same input, same output, every time. You can backtest routing rules on historical data and know exactly what your accuracy and escalation rate will be before going live

Try it

The margin field ships with every prediction. No extra parameters needed

# Try it with any classifier
curl -s -X POST https://playground.jeffyclassify.com/v1/predict \
  -H "Content-Type: application/json" \
  -d '{"task":"clinc_oos","text":"I need to book a flight to New York"}'

# Or locally
pip install jeffy-classify
jeffy-serve

Available classifiers for routing:

• clinc_oos: 151 intents across 10+ industries, plus out-of-scope detection
• banking77: 77 banking customer service intents, 94.3% accuracy
• massive_intent: 60 intents, works in 51 languages
• sms_spam: binary spam detection, 99.1% accuracy

Or train on your own data in seconds