Mid-size local models are now competitive for AI Agents!

A year ago, getting useful work done with open-weight LLMs on consumer devices was mostly aspirational. Useful models lived in the clouds of trillion-dollar AI companies behind metered and rate-limited APIs — local stacks were hobbyists’ weekend projects.

Last weekend, I benchmarked the latest mid-sized & small open-weight models on my GB10 box (DGX Spark family, 128Gb unified RAM) and live-tested the best ones with OpenClaw. Consumer grade devices like Mac Mini, Mac Studio and the GB10 family tend to be memory bandwidth constrained — they can only deliver acceptable speeds for interactive use from a few billion to a few tens of billion active model parameters per inference which is the space of mid-sized & small models. While running locally has major cost & privacy benefits, the big question is are these small & mid-sized models good enough? I tested on speed, simple problem solving, instruction following & medium complexity programming needed typically by personal AI agents (aka Claws) and compared them with frontier “flash” models from cloud providers.

I found that the picture has dramatically shifted. Certainly, modern mid-sized open-weight models haven’t quite caught up to frontier models yet — that gap is real. The gap, however, is now sufficiently narrow that for many practical use cases like “Claw” assistants & most types of automation, they are genuinely competitive and practical. Mid-sized open-weight models can reduce operating costs by 10x-20x (essentially just electricity bill) and deliver a massive privacy benefit — private data, medical records etc. don’t leave your network. Only very complex skill development absolutely requires a hot swap to a frontier cloud model.

Here’s what I found, what the surprises were, and how the trade-off shakes out as of May 2026.

My picks for DGX Spark Family (May ’26)

Best local model: Qwen 3.5–122B A10B-MoE-NVFP4-MTP-Sehyo

This massive model checkpoint from the Qwen 3.5 family just about runs with acceptable speed on the GB10 thanks to speculative token prediction (MTP) while closely approaching (and occasionally beating) some frontier flash models. The 122 billion parameter network shows its value in the medium complexity programming benchmarks & it’s now been my OpenClaw daily driver model for a week. I also expect it will be a good-enough backup for coding agents when I blow my quota.

Best lean local model: Qwen 3.6–35B A3B-MoE-NVFP4-MTP-RedHatAI

While all model checkpoints from the Qwen 3.6 family performed in a tight band, this version featuring NVIDIA’s new NVFP4 quantization that reduces memory footprint and increases speed can run on just 36GB free RAM. This makes this high-quality model solidly viable for higher-end Mac Minis & Pros. I now switch to this as my backup model for OpenClaw whenever I want more free RAM on my GB10 to run other Machine Learning experiments or benchmarks.

Best value cloud model: Gemini 3.1 Flash-Lite & Qwen 3.6 Flash

This was a close call. Gemini 3.1 Flash-Lite (73 overall) and Qwen 3.6 Flash (74) finished neck-and-neck on cost/performance, and both were a surprise — competitive even amongst cloud models, blazingly fast, and crucially both ship with strong data protections by default, unlike DeepSeek V4 Flash. DeepSeek edges them on raw price, but the privacy defaults make these my preferred fallback cloud models.

Top Three Takeaways

1. Mid-sized models are good enough for most agentic tasks, small models are still mostly hype

A single mid-sized model running on a desktop-class box is now reliable enough to power everyday AI agents for the bulk of real work — answering questions, following instructions, writing and running skills. Small models still stumble, especially on medium complexity code generation, making too many errors to trust unsupervised. “Mid-sized” is the sweet spot where quality crosses the usefulness threshold. The savings and privacy benefits are not marginal — my own daily cost dropped from $10–12 a day to just the extra cost of electricity — the hardware pays for itself inside a year.

2. Data-center grade open-weight models are competitive for enterprise automation

Just as a model running on a box under your desk is good enough for everyday work, larger cousins of the same models running on cloud GPUs become a serious option for enterprise. Indeed, the top scoring flash model in this test is an open-weight model. Today most routine AI tasks are sent to expensive frontier APIs by default. For the long tail of internal tools and assistant workloads where “good enough” really is good enough, open-weight models on GPU servers can do the job at a lower cost — and fully retain control of data.

3. Barring reaching AGI, commoditization may shake up the AI industry over the next two years

Frontier providers are raising prices and tightening usage restrictions, yet the math suggests they may be earning extraordinary margins on inference. As open-weight alternatives become “good enough” for most work at a fraction of the cost, that pricing power might become hard to defend. Barring a genuine breakthrough — like true general intelligence that only the largest models can deliver — commoditization could dramatically reshape who captures the value in AI.

The Numbers

Press enter or click to view image in full size

Model bench-marking results.

Technical Insights

1. Larger models still dominate complex code generation and debugging

Complex coding — writing a multi-file change and debugging it when the tests fail — is where extra parameters of large model show their value. Our local heavyweight, Qwen 3.5–122B A10B — with the most parameters (122 billion) outperformed even several frontier cloud flash models and smaller models of the next Qwen 3.6 generation. This results in a much more reliable AI Agent that requires less babysitting.

2. Mixture of Experts (MoE) models are surprisingly competitive against Dense models.

Intuition says that a Dense model, which fires all its weights on every inference, should beat a Mixture of Experts model that fires only a fraction. In this test however, Dense and MoE models within the same generation landed close. The Qwen 3.6 family models landed dead even at an overall score of 61. The Gemma-4 Dense model only slightly outperformed its MoE cousin. MoE models therefore win on practicality, since firing a fraction of the weights makes them much faster on bandwidth-constrained consumer hardware.

3. Dense models can’t deliver acceptable performance on consumer hardware without speculative prediction

Because they fire every weight, the dense models only deliver acceptable interactive performance once a speculative-prediction drafter is bolted on. With one, Qwen 3.6–27B-Dense meets my 20 tok/sec comfort threshold and the new Gemma 4–31B Dense lands just under it at 18 — just about usable despite their size. Without speculative prediction, a no-drafter Gemma 4–31B barely made it to mid-single digits and wasn’t included in testing.

4. The 4.5 effective bits NVFP4 quantization delivers surprising quality, speed & memory advantages

AI Labs generally train and ship new models at 16-bit precision, which puts our 122-billion-parameter Qwen 3.5 at 238 GB just for the weights — squarely data-center territory. NVFP4, natively supported on modern NVIDIA hardware, squeezes each weight down to roughly 4.5 effective bits with very little trade-off on quality. In testing, the NVFP4 quantized Qwen 3.6–35B-A3B landed in a dead heat with its 8-bit FP8 cousin. A model quantized with NVFP4 fits and runs fast on bandwidth-constrained consumer devices — for almost no quality cost.

5. While the cloud frontier is very competitive, Alibaba dominates in mid-sized models

Among cloud flash models the field is tight — Kimi, Gemini, Alibaba, Anthropic and DeepSeek keep trading the lead. But on the mid-sized models you can run locally, Alibaba’s Qwen is in a class of its own (circa May ‘26): the comparable releases from NVIDIA, Google and Mistral AI didn’t come close, making too many errors on multi-step agentic work to trust unsupervised. Small models of which we tested two are only able to reliably handle straightforward problems.

P.S. First post here in a while. I’ve been investing fair bit of time in agentic AI recently. As I have learned quite a bit from experimenters pushing the frontier of open models, I thought I’d share back my personal learnings. More to come.

¹ Overall Score — weighted blend of coding quality, instruction following capability and speed; higher is better.

² Cost — cloud model cost calculations blend input, output & cache-hit rate benchmarked from observed OpenClaw turns across a variety of tasks (60k input, 500 output, 75% cache hit). Costs are expressed relative to the lowest cost cloud flash model here — DeepSeek v4 Flash

³ Speed — tokens per second per user, how snappy the model feels in an interactive assistant loop. Cloud figures are end-to-end including network latency transit. Cloud models run on very powerful servers and tend to be fast but have a latency to generate first token

⁴ Code Correctness — Measures whether short functions the model writes work as a good proxy for the kind of off the cuff actions that Agentic Assistants take. Average pass rate on HumanEval+ and MBPP+ (EvalPlus, greedy temperature=0).

⁵ Instruction Following — accuracy on IFEval, a benchmark that checks whether the model obeys explicit constraints in a prompt (format, length, content rules). Proxy for how reliably it follows directions.

⁶ Coding 1st Try — pass rate on the Aider polyglot coding benchmark on the first attempt. Measures whether the model can complete a realistic multi-file coding task in one shot. Python and Javascript which are the typical Agentic Assistant languages were evaluated

⁷ Coding 2 Tries — same Aider benchmark, allowing one retry after seeing test failures. Measures whether the model can self-correct, which is closer to how an agentic assistant works.

Appendix 1: Understanding model names

Qwen 3.5–122B A10B-MoE-NVFP4-MTP-Sehyo

Qwen 3.5 — the model family and version
122B — total parameters in the network in billions. A rough proxy for how much the model “knows.”
A10B — active parameters per inference in billions; In a Mixture-of-Experts (MoE) model only a fraction of the weights fire for each inference which improves speed and reduces memory footprint.
MoE — the model architecture — in this benchmarking Dense or Mixture of Experts; in dense models, all the parameters fire for each inference which hits against memory bandwidth walls on consumer devices.
NVFP4 — the quantization format — NVIDIA’s 4.5-effective-bit floating point that compresses weights so the model fits in far less memory and runs faster with a little quality loss. Other common formats include FP8 (8-bit floating point, weights memory ≈ params) and BF16 (16-bit floating point, weights memory ≈ 2X params). Models are generally orignally trained and released at BF16 (although DeepSeek notably trains at FP8 instead) and then quantized.
MTP — speculative token prediction mechanism — in this case Multi-Token Prediction — a native architecture where the model predicts several future tokens at once, boosting tokens/sec.
Sehyo — the checkpoint author/publisher — the person or org on Hugging Face etc who produced this specific quantized release.