Abnormal classifies billions of emails a week, and in this post we test how decision models hold up against the recent hype, using our own non-public email security data.
Our production models are behavioral, learning what normal looks like for each sender, recipient, and organization from signals like sender history, authentication, and the org directory. Deciding whether that request makes sense between this sender and this recipient takes understanding the text itself and knowing how businesses normally operate, like whether a vendor would really change bank details in the middle of an invoice thread. A model built on engineered features only sees the signals it was given. An LLM has learned language and that kind of everyday business knowledge from pretraining, so it can read the request the way an analyst would, with the same behavioral signals as context
Decision models (often) read the email and every question in one forward pass and return a probability for each option without generating a token, so an answer costs about what it costs to read the input. We think that makes LLM-class judgment cheap and fast enough to deploy at far larger scale, behind the right prefilters. Each question is also written in plain language, so a new detector can start zero-shot as a written question rather than a feature to build and a model to retrain.
We tested TypeSafe’s Jev, the OpenAI Decisions API, 34 open decision models we self-hosted, and four frontier LLM configurations (40 models in all)1, then fine-tuned one of the small open models ourselves.
The dataset is 1,985 non-public, fully synthetic emails with no customer data, modeled on the hard cases we see in production (attacks and the tricky false positives that look like them). Rather than prompting a model to write a phishing email, we generated each one to match the class, attack type, and detection signals of a real hard case without reusing its identifiers.
Every model gets the same email, rendered as the fields an email security system extracts, and answers three questions in one request: is it an attack (yes or no), what kind of mail is it (attack, spam, graymail, or safe), and for attacks, which type (phishing, BEC, scam, malware, or recon). Decision models read the record natively, while LLMs get it as JSON and have to return a probability for every option under a strict schema.
To rank models on one number, we use an email score, defined as 100 × the mean of verdict macro-F1, attack-type macro-F1, and attack recall at 95% precision. The terms track three jobs a detection system does (sorting the mail, naming the attack, and catching attacks while keeping false alarms rare). Macro-F1 counts rare classes like malware as much as common ones, and the equal weights mean no model scores well by acing just one job2.
On cost versus the composite email score, larger models generally score higher and the frontier LLMs lead. Those gains are rarely cost-effective, though, since each step up the curve costs far more than the one before it. Jev and the OpenAI Decisions API both sit on the Pareto frontier, scoring well above the open models at their price3.
In email security, the composite score isn’t what decides whether a model is usable. When a model calls an email an attack, we pull it from the inbox, so every false positive is a legitimate and often important email that a customer never sees. Precision is the constraint and recall is the prize, so we look at each model’s precision-recall curve and read off how many attacks it catches at 99% precision.
Against cost, recall at 99% precision collapses the frontier to a handful of models. Most of the field, including every open model we self-hosted, catches only a small fraction of attacks, and Jev stays on the frontier on price alone. The OpenAI Decisions API is the standout, catching most of what luna-6-max does at a fraction of the cost and latency. The most instructive miss is luna-6-low, the same luna-6 model prompted as an LLM. It matches the Decisions API on average precision but falls far behind at 99% precision, because threshold-free metrics hide the part of the curve you actually operate in.
Security systems often act on the probability, not just the label. Above one threshold an email is remediated automatically, a middle band goes to review, and everything below is delivered. Those thresholds have to keep meaning the same thing across tenants and model upgrades, and the probability is also what lets one detector's output be combined with others or weighed against the cost of a miss. All of that only works if a stated 0.9 means about 90% of those emails really are attacks.
The decision APIs come back among the best calibrated, with Jev the best calibrated of any model we tested. The frontier LLMs land in the middle even when the prompt asks them directly for honest, calibrated probabilities. The small open decision models are the worst, base kev-4b among them, so calibration depends on the model rather than the interface.
As an experiment, we fine-tuned kev-4b on a variety of email and non-email classification tasks, a single-epoch LoRA run over 60K examples that cost about $200 on L40S GPUs. Nothing in training overlaps with this benchmark in either emails or task format, since the benchmark’s synthetic emails were never trained on and its text encoding and question wording differ from ours4.
The fine-tune breaks the Pareto frontier, lifting recall at 99% precision from 7% to 66%, higher than any model we tested at any price5. Calibration improved with it, going from one of the worst in the field to among the best.
Attack-type macro-F1 fell from 68 to 37 in the same run, even as attack detection jumped, and the model now labels most BEC and recon emails as scams. Our training mix covered the four-way verdict but never attack types, so we suspect the fine-tune sharpened the question it was trained on and degraded one it wasn’t.
Training and evaluation leave out what it takes to serve a model online at our scale, which comes down to where it can run, what it really costs, and whether the GPUs exist.
Data residency. EU mail needs to be processed in the EU. An API only works if it can serve there, and a self-hosted fleet has to exist in every region we operate in, not just wherever GPUs are cheapest.
Real serving cost. The cost charts assume every GPU is busy all the time. Priced from kev-4b’s measured throughput in our harness at on-demand GPU rates, serving it costs more than the chart shows even with no idle time. Mail follows the workday and peaks need headroom, so actual utilization is lower and the cost higher still.
GPU supply. At our volume, running a 4B model on every email takes a GPU fleet that’s hard to get even with the budget for it.
Today, the equivalent detection in our online stack costs about a cent per thousand emails, on a mix of CPUs and GPU accelerators6. Decision models moved LLM-class judgment from dollars per thousand emails to cents and from seconds to milliseconds, which puts it within reach for the slice of traffic a prefilter sends them but not for every email.
We expect the right architecture at this scale to be a careful mix of model sizes and types, with cheap models on everything, decision models behind prefilters, frontier LLMs on the few emails that need real reasoning, and feedback loops that let the system improve itself.
If you want to help build it, we’re hiring.