Smarter Log Triage: Why Cheap Flash LLMs Master SOC Context Discipline

We benchmarked flagship LLMs against cheap Flash models on L1 log triage. Here is what your API bill actually buys.

Executive Summary

Security teams are routinely sold “frontier models for the SOC.” We asked a narrower, money-relevant question: On short first-pass log triage, do expensive flagship models beat cheap Flash/Nano models by enough to justify the bill?

We built SOC Log Triage (LT): 22 synthetic log bundles (8 minimal-pair twins + 6 controls) evaluated against four gold labels (attack, misconfig, noise, needs_more_data), paired with a separate panic trap measuring false alarms on non-attack events.

The 30-Second Takeaway

The Delta That Matters: Top expensive models (Claude Sonnet 5, GPT-5.6 Luna) score 0.961 LT, while top cheap models (Gemini 3.8 / 3.7 Flash) score 0.894 LT — a modest +0.067 (~7%) performance gap.

False Alarm Parity: Cheap Flash models match flagships on panic trap calmness (1.000 score), showing no tendency to overreact to scary strings like /etc/passwd.

More Expensive ≠ Better: Flagships like GPT-5.5 (0.571) and GPT-5.6 Sol (0.793) were routinely beaten by Flash models due to context discipline failures or output formatting collapses.

The Policy Rule: Route high-volume L1 triage to Flash/Nano models. Pay for top-tier frontier models only on escalated items where a 7-point accuracy bump is worth 10x–20x higher API costs.

1. The $10/M Token SOC Trap

An L1 SOC night shift is not “find a CVE in a repo.” It is a relentlessly fast stream of noisy evidence packs:

• Tens of thousands of /wp-admin HTTP 404s
• An AccessDenied spike directly following a CI/CD deploy
• A truncated POST /api/transfer line missing its HTTP status code
• Kube-probe hitting /healthz every 10 seconds

When you paste these into an uncalibrated LLM and ask “Is this an attack?”, many models panic on scary tokens. That isn’t threat detection — it is alert fatigue with a chatbot face, costing $10+ per million tokens.

Public LLM leaderboards miss this nuance. Coding benchmarks ask “Did you find a bug?” Chat benchmarks ask “Are you helpful?” Neither asks:

1. Did you separate attack from misconfig from noise?
2. Did you abstain when the shipper truncated the log line?
3. Did paying 10x more buy any real accuracy?

SOC Log Triage is a diagnostic probe. Small dataset sizes with minimal-pair twins allow single context-discipline failures to register clearly, showing whether a model reads control facts or merely acts on keyword impressions.

2. Testing Context Discipline, Not Token Panic

We borrow the minimal-pair twin discipline from linguistics: two log lines that share scary surface features, where exactly one control fact flips the gold label.

The Gold Taxonomy

• attack — Evidence of active exploitation or compromise in progress.
• misconfig — Config/IAM/CORS/debug errors following a change without active exploitation.
• noise — Expected background traffic (health checks, blocked generic scanners).
• needs_more_data — Incomplete or truncated logs. Forces abstention instead of guessing.

3. Dissecting Minimal Pairs: The Twin Method

The dataset (N=22, frozen in git) contains 16 minimal-pair twin items (8 pairs) and 6 unpaired control anchors. Here is how tiny control facts flip the diagnosis:

Deep Dive 1: Web Reconnaissance (twin_wp_probe)

• Shared Surface: Requests targeting /wp-login.php, /xmlrpc.php, and /.env.
• Attack Variant: Distributed requests originating from multiple external IPs (203.0.113.*, 198.51.100.*). Gold: attack
• Noise Variant: Requests originating from a single egress IP (34.102.136.180) with User-Agent IridiumCI/1.0 and a comment noting a scheduled scan. Gold: noise
• Why it matters: The path list alone is not a compromise. A model that labels both attack failed context reading.

Deep Dive 2: Cloud IAM Escalation (twin_iam_denied)

• Shared Surface: AccessDenied events targeting sensitive bucket prod-invoices.
• Misconfig Variant: Errors occurring immediately after an automated deploy-bot policy update removed s3:GetObject. Gold: misconfig
• Attack Variant: Unknown OIDC session (MFA=false) executing successful bulk s3:GetObject downloads following an initial denial. Gold: attack
• Why it matters: AccessDenied after a deploy bot policy tighten is a post-deploy outage. Bulk GetObject after a shady AssumeRole is exfiltration.

Summary of Remaining Minimal Pairs

• twin_api_post (API Requests): Shares POST /api/v1/transfer. Flips to attack on sqlmap User-Agent + 200 OK + SQLi body. Flips to needs_more_data when status and body are truncated (upstream closed).
• twin_ssh_auth (SSH Bastion): Shares failed SSH passwords. Flips to attack on Fail → Accepted → passwd root → lateral hop. Flips to noise on invalid-user brute force from internet background IPs.
• twin_app_500 (App Error Spikes): Shares checkout HTTP 500 bursts. Flips to attack on SQL syntax errors with UNION SELECT in parameters. Flips to misconfig on connection refused aligned with a DATABASE_HOST deploy.
• twin_waf_block (WAF Events): Shares security events on /admin. Flips to attack on bulk export after an out-of-band password reset. Flips to noise on standard WAF BLOCK of generic zgrab scanners.
• twin_s3_public (Public Buckets): Shares public S3 GETs on asset buckets. Flips to attack on high-volume downloads of /private/payroll.csv. Flips to misconfig on fetching /branding/logo.png after an ACL flip.
• twin_vpn_login (VPN Sessions): Shares VPN auth for valid users. Flips to attack on MFA bypass + impossible physical travel (US to KZ). Flips to needs_more_data when telemetry regarding device state is missing.

4. Benchmark Results

What Does Your Money Actually Buy? A 7% Gap.

Top-tier flagship models hit 96% accuracy on log triage, while cheap Flash models hit 89%.

That 7-point difference comes down to missing just 1 or 2 subtle edge cases out of two dozen complex scenarios. While flagship models are technically sharper, paying 10x to 20x more for every single log in your queue is like driving a supercar to drop off the mail.

5. Panic Trap: False Alarms Are Already Solved

When asked directly “Is there a confirmed active attack or compromise right now?” on the 10 noise/misconfig items, Flash/Nano models and expensive flagships both score 1.000.

Paying more does not buy calmer behavior. Modern lightweight models no longer scream “attack” at every kube-probe or blocked vulnerability scanner. The separation between model tiers lives entirely in fine-grained taxonomy classification and explicit abstention on truncated logs.

6. Actionable SOC AI Engineering Playbook

Security engineering teams should structure LLM triage pipelines around tiered routing rather than defaulting to flagship models:

                 [ Raw Ingest Log Stream ]
                              │
                              ▼
                 [ Step 1: L1 Triage Filter ]
                   Model: Gemini 3.8/3.7 Flash
                   Cost: ~$0.10 / M Tokens
                              │
            ┌─────────────────┴─────────────────┐
            ▼                                   ▼
[ High Uncertainty / Truncated ]       [ High Confidence ]
  LT Label: `needs_more_data`            LT Label: `noise` / `misconfig`
  or Low Classification Margin            or Clear `attack`
            │                                   │
            ▼                                   ▼
[ Step 2: Escalated L2 Audit ]         [ Auto-Route / SIEM Action ]
  Model: Sonnet 5 / GPT-5.6 Luna       Log to SIEM / Silent Drop
  Cost: ~$3.00–$15.00 / M Tokens

Three Rules for SOC Buyers

• Default to Flash: Route 100% of initial log volume through Flash/Nano models.
• Escalate by Exception: Reserve Claude Sonnet 5 or GPT-5.6 Luna for the hard slice where logs emit needs_more_data or high ambiguity.
• Audit Vendor Pitches: Never buy “security AI” on a chat demo of one scary line. Demand twin-based triage benchmarks and false-alarm scores.

Benchmark Resources & Open Source

The complete dataset, evaluation harnesses, and diagnostic scripts are fully open-source:

• GitHub Repository (MIT): mziqudhd92/kaggle-soc-log-triage-benchmark
• Kaggle Benchmark (Label Task): log-triage-label
• Kaggle Benchmark (Panic Trap): log-triage-panic-trap
• Companion Code-Patch Study: ART Benchmark