I've taken the deep dive into local AI so you don't have to. This is how to use AI that works on your local PC or Mac. No cloud.
My feed is filled with local AI stuff. There's a lot of "boy who cried wolf" feeling around it because I hear claims and then only later you find out they were running on $100k of hardware. For that I could subscribe to Claude Max, Codex Pro and Grok Super Heavy Build for decades.
What I really wanted to know is what hardware can do what. What were the real limitations and what they could really do. And so I dug in. And now, you can benefit from that journey. And yes, I use em-dashes — a lot. They're mine. AI can't claim them.
Part 1: The AI Models
AI Parameters: 2B, 3B, 9B, 27B for fun and profit
When you see a model called "Llama 8B" or "Gemma 27B," the number is its parameter count, the number of learned weights inside the neural network, in billions. Parameters are, roughly, the model's capacity to know things and reason about them.
Rough breakdown by size:
~1B and under: Autocomplete with hallucinations of grandeur. Fine for classification, summarizing a paragraph, simple formatting tasks.
2B–4B: Genuinely conversational. Can follow instructions, answer general questions, and with careful prompting, do simple tool use. This is the sweet spot for phones and NPUs.
7B–9B: The workhorse class. Decent general knowledge, can follow multi-step instructions, handles light agentic work (call a tool, read the result, respond).
13B–30B: Where local models start feeling like "real" AI. Better judgment, fewer hallucinations, can recover from their own mistakes mid-task.
70B+: Approaching frontier-model territory, but you need serious hardware. Think a Mac Studio with 128GB of unified memory or a multi-GPU rig, not a laptop.
Parameter count also affects speed, not just smarts. Every single token the model generates requires reading essentially all of the parameters from memory. A 27B model isn't just smarter than a 3B model, it's also about 9x more work per word. Keep that in mind; it becomes a big deal when we start thinking about what you can do in real-time vs. what you should schedule out.
Quantization
Models are trained in 16-bit precision. An 8B model at 16 bits per parameter is 16GB just for the weights.
Think of it like graphics formats. 16-bit precision is like a bitmap. No one, anymore, sends a .BMP in email.
Quantization is lossy compression for neural networks: store each weight in 8, 5, or 4 bits instead of 16. So it's like taking that BMP and turning it into a JPEG. At 8-bit quantization it's essentially the same as the original uncompressed version. At 4-bit it's lost some but probably not enough to matter other than in benchmarks. Below that and it starts to get pretty bad. 4-bit is generally the sweet spot.
The alphabet soup: Q4_0, Q4_K_M, Q8_0, IQ4_NL
Quantization also has many ways of doing it. Just like there's a ton of graphic formats there's a ton of ways of quantizing these models, each with their own trade-offs.
In my experience, I tend to focus on MLX (for Mac hardware), the ones that NVIDIA likes, and the ones that NPUs from Qualcomm and others will like.
A sampling:
Q8_0: 8-bit. Nearly lossless, twice the size of 4-bit. Worth it only for small models where the size doesn't hurt (a 0.6B model at Q8 is under 1GB, so why not).
Q4_K_M: the modern 4-bit "K-quant." Smarter allocation of bits (important layers get more precision). Usually the best quality per gigabyte, and the default recommendation on CPUs and GPUs.
Q4_0: the original, simplest 4-bit format. Slightly worse quality than Q4_K_M, but its plain block layout is what specialized hardware paths are built for. On ARM CPUs and NPUs, Q4_0 is often dramatically faster than Q4_K_M because the fast kernels only speak Q4_0.
IQ4_NL: a newer 4-bit format using a non-linear codebook. Great quality, but hardware support is spottier.
The format has to match the hardware. In our own testing, the same 4B model ran at 37 tokens/second as Q4_K_M on CPU but collapsed to 14 when routed to the NPU, because the NPU path couldn't handle K-quants and fell back to a slow path. Same model, same computer, same "4-bit," 2.5x difference. This is the single most common way people accidentally sandbag their local AI setup. This is why it's still a headache and programs like Clairvoyance are exploding in popularity — they just take care of this nonsense.
Part 2: Hardware constraints
TOPS, NPUs, GPUs: the speed of thinking
Chip vendors advertise TOPS, trillions of operations per second. A Copilot+ certified laptop's NPU claims 40–80 TOPS; an RTX 4090 delivers over 600. You'd think a 45-TOPS NPU runs AI at some meaningful fraction of a 4090's speed.
It does not.
There are two very different phases when a model responds to you:
Prefill: reading your prompt. This can be done in parallel so GPUs and NPUs do great here.
Decode: writing the answer. Tokens come out one at a time, and each one requires streaming the entire model through memory again. TOPS are nearly irrelevant; memory bandwidth is everything.
So a laptop NPU can read and understand a full article of text in half-a-second. But actually commenting on it is limited by the memory bandwidth which might take a minute.
RAM: system RAM, GPU VRAM, and unified memory
The model has to live somewhere, whole, in fast memory:
Discrete GPU (VRAM): Fastest option by far, but VRAM is scarce. A 12GB card fits a 9B comfortably or a 27B not at all. Spilling layers to system RAM works but every spilled layer runs at system-RAM speed.
System RAM (CPU): Plentiful and cheap (32GB fits anything you'd sanely run) but slow, see below.
Unified memory (Apple Silicon, Snapdragon X, AMD Strix Halo): CPU, GPU, and NPU share one pool. The great trick of a 128GB Mac is that all 128GB of it is available to the model at decent bandwidth, even though that bandwidth isn't remarkable on its own.
Sizing rule of thumb: model file size + 20–30% for the working context. A 5GB Q4 model wants roughly 7GB free.
Memory bandwidth: the often-ignored bottleneck
Memory bandwidth predicts local AI decode speed better than any other spec:
Typical laptop DDR5: ~60–90 GB/s
Copilot+ certified laptop (LPDDR5X, shared): ~120–152 GB/s
Apple M5: ~153 GB/s
Snapdragon X2 Elite Extreme (LPDDR5X, shared): ~228 GB/s
Nvidia DGX Spark (LPDDR5X, unified): ~273 GB/s
Apple M5 Max: ~614 GB/s
Radeon RX 9070 XT (GDDR6): ~645 GB/s
RTX 4090 (GDDR6X): ~1,008 GB/s
RTX 5090 (GDDR7): ~1,792 GB/s
RTX PRO 6000 Blackwell (GDDR7): ~1,792 GB/s
These speeds make the difference between whether you should be doing a task in real time or be scheduling it.
One of the first things I realized is that my laptop of choice can run a 27B model just fine. It's just slow at doing it. But most of the work I need to do with AI is not real-time. The very first feature that I used in Clairvoyance was the scheduling. I have so many dashboards, crash reports, sentiment reports, sales data coming in that I just have it run overnight for me to look at in the morning. I used to have that on Claude and that stuff was costing me $100 a month in tokens. But a 27B model can do it exactly as well and at no cost as long as I schedule it.
Diversion: Geek out on Hardware
The RTX 5090 is the current consumer king. Nearly 1.8 TB/s means a 5GB model decodes at hundreds of tokens/second, and its 32GB of VRAM fits a 4-bit 27B with room to spare. If your goal is "fast local AI, money is no object," this is the answer. As of this writing, an RTX 5090 currently retails for a little over $4.6 trillion dollars.
The RTX PRO 6000 Blackwell is the 5090's workstation sibling: same ~1.8 TB/s bus, but 96GB of VRAM, enough to hold a 4-bit 70B (or an 8-bit 70B, barely) on a single card at full speed. It solves the problem two 4090s can't (see below), big model and big bandwidth in one memory pool, for roughly the price of a decent used car. The use case for us would be you'd park one of these on the rack and have several users on it running a 27B model in real time.
The Nvidia DGX Spark is slower than you'd expect. It's marketed as an AI supercomputer for your desk, and for capacity it delivers. 128GB of unified memory fits models no consumer GPU can touch. But its 273 GB/s bandwidth is less than half an M5 Max, a quarter of a 4090. It runs big models acceptably; it does not run any model fast. You're buying capacity, not speed. There's a reason these are in stock at Microcenter. I'd just buy a Snapdragon X2 Elite Extreme or Mac M5 Max instead.
AMD's Radeon RX 9070 XT at ~645 GB/s out-decodes every laptop and more than doubles the Spark at a fraction of the price. It's one of the best value plays in local AI, provided the model fits in its 16GB. A 4-bit 9B flies, a 4-bit 27B just squeaks in. The main issue here is the amount of RAM they put on it. Envision me shaking my fist at my friends at AMD. 16GB max? Why? Why did you do this?
Two 4090s do not make a 2,000 GB/s machine. The usual way to split a model across two cards is by layers, half the network on each, and a token still passes through the layers in sequence, so each card sits idle half the time. What you actually buy with the second card is capacity: 48GB of VRAM, enough for a 4-bit 70B, at roughly single-4090 speed. (Tensor-parallel setups can claw back some speed, but that's server-software territory, not a checkbox.) Two mid-tier cards to "add up" bandwidth is a common and expensive misunderstanding. But 1000 GB/sec is no joke. 48GB of RAM handles that 27B model just fine.
Extra Detail Stuff
The back-of-envelope math: decode speed ≈ bandwidth ÷ model size. A 5GB model on a 152 GB/s bus tops out around 30 tokens/second no matter how many TOPS you have, because generating each token means reading all 5GB. Our measurements land right on this line: 9B models at 4-bit decode at 21–23 tokens/second on a Copilot+ certified laptop whether we use the CPU, the NPU, or both.
This is also why we learned, the hard way, with a stopwatch, that routing big models to an NPU is pointless or worse. The NPU shares the same memory bus as the CPU, so it can't decode any faster, and its dedicated fast memory is tiny, so anything beyond roughly 3B parameters doesn't fit where the NPU is actually fast. Our working rule is now simply: models over ~3B run on the CPU; the NPU is for small models and for prefill. Small model on NPU: brilliant (a 3B doing a full task in 5.8 seconds). 9B on NPU: same speed as CPU at best, sometimes slower.
Smaller models are proportionally faster, not just a little faster. Half the parameters, twice the tokens per second, on the same machine. Which brings us to what you should actually run.
Part 3: Real world usage
The right model for the right task
"Which model should I use?" is the wrong question until you've answered "for what?" Capabilities don't scale evenly with size. They arrive in tiers:
Chat and Q&A: Works surprisingly far down. Look for a 3B model. That's your sweet spot for this. As soon as you go to 4B you max out the Copilot+ spec for memory bandwidth. TURN OFF thinking if the model supports it. You lose all the benefit of speed and a 3B model will never think itself into being smart.
Tool use (the model calls functions: search, open a file, run a query): 9B. This is the "Edit this email" level.
Agentic work (multi-step: search, read the result, decide, act again): Right now I would say 27B is the sweet spot. But a 13B model can do a lot of this and run real-time on an M5 Max level machine.
Judgment (which of these is better? is this claim supported? did I make a mistake?): The last thing to emerge. In our sweeps, the 27B was the only model that went 3-for-3 on every configuration with clean or self-correcting tool use. It noticed its own errors and fixed them. That is a capability, and it doesn't compress. But this is only on newer 27B models.
Side Note: Thinking is not always a good idea
Reasoning models, the ones that deliberate in a visible scratchpad before answering, look like the obvious way to buy quality without buying parameters. At the small end it backfires, and our own benchmark testing showed it was pretty terrible.
We ran our favorite 3B model (VibeThinker-3B) through the same find-and-display task as everything else here. With thinking nominally disabled, its reasoning training still leaked into the output: 2,000–3,000 generated tokens per run, against roughly 70 for a conventional model on the identical task. At 43 tokens/second, that's about a minute of deliberation before the useful answer starts. End to end it took 85–104 seconds, and it was unreliable. One run never called the tool at all. A conventional 9B, three times the parameters, finished the same task in a sixth of the time.
Then we gave it a prompt template that actually suppresses the deliberation. Same model, same hardware, same task: ~56 generated tokens, 5.8 seconds end-to-end including a cold model load, and three successes out of three across CPU, hybrid, and NPU. It went from the worst configuration we had measured to the best one in the sweep. Thinking != Smarter.
Below roughly 4B, turn thinking off. If a task genuinely needs deliberation, those tokens are better spent on a bigger model answering plainly. The 27B above did its self-correcting with thinking switched off. One distinction worth keeping straight: a model distilled from a reasoning model is not the same animal. The 4B distill in the same sweep emitted 155 tokens per task rather than 3,000, and was one of the fastest reliable configurations we measured.
Time to first token
The first speed you feel is the pause before anything appears. It's the sum of model load (if not already resident, loading 5GB off an SSD takes seconds) plus prefill of your prompt. This is where NPUs and GPUs matter. Now you know why that first "Hello" takes so long. It's basically booting the model.
Prefill speed
Measured in tokens/second of input processing, and the spread is huge: we've measured the same 3B model prefilling at 576 tok/s on CPU and 1,908 tok/s on the NPU. For chat, with short prompts, you barely notice. For anything agentic, where every turn re-feeds the growing conversation plus tool results, prefill speed compounds and quickly dominates. This is the legitimate use of that big TOPS number.
Tokens per second (decode)
The number everyone quotes, and the one that governs how it feels once text is flowing. Real measurements from a Copilot+ certified laptop, all 4-bit unless noted:
0.6B (Q8): ~110 tok/s — Instant
3B: 43–57 tok/s — Faster than you read
4B: ~37 tok/s — Fast
9B: 21–23 tok/s — Comfortable reading pace
27B: ~8 tok/s — Watching someone type
Note how cleanly it tracks model size. That's just the memory bandwidth ceiling from Part 2 showing up in practice.
Real-world examples: 3B for chat, 9B for light tool use, 27B for real work
Putting it all together, from our own benchmark sweeps (same task, find a document and display it, run across 35 hardware/model combinations):
3B on NPU: completed the entire task in 5.8 seconds end-to-end, including starting the server and loading the model cold. Warm, the same turn takes 2.4 seconds. Reliable across every run. This is the "it just feels instant" tier, and it's what a modern AI laptop should be doing for quick tasks.
9B on CPU: ~18–24 seconds for the same task. Noticeably more thoughtful answers, comfortable with tools, still fast enough that you don't context-switch away.
27B on CPU: ~41 seconds. Slow enough that you go do something else, but it was the only model that never failed, and the only one that caught and corrected its own mistakes. For work where being wrong costs more than waiting, this is the one.
So use the fast tier for things you're sitting there waiting on, and the slow one for things you hand off. Chat with the 3B, give the 27B a job and come back later.
And "real work" is not a euphemism anymore. The 27B in that benchmark is Qwen3.8-27B, released in mid-August 2026 under Apache 2.0, and its published numbers are the kind that would have been science fiction for a local model a year ago: 61.7% on SWE-Bench Pro and 70.7% on CoWorkBench, the latter edging out the 68.2% Alibaba reports for Claude Opus 4.6 Max, a frontier model. The pattern across independent write-ups is consistent: the 27B leads on agentic software-engineering benchmarks, while the frontier model keeps its lead on pure-knowledge tests (GPQA Diamond, Humanity's Last Exam) and raw terminal coding. Two caveats before you cancel anything: those are vendor-reported scores at full precision, and the 4-bit quant you'll actually run gives some of it back. But directionally, the gap between "toy" and "frontier" has collapsed to a benchmark-by-benchmark argument, for a model that fits on a gaming GPU.
But how many tokens per second do we actually need?
Useful anchors:
People read at roughly 5 tokens/second (~250 words/minute). Anything above ~10 tok/s outruns your reading for chat.
For agentic work the bar is higher, because most generated tokens are tool calls and reasoning you never read. You're waiting on the outcome. There, 20+ tok/s is where waiting stops being painful, and below ~10 it's genuinely tedious.
What does Claude Code do, as a reference?
For calibration against the frontier: Claude Code, Anthropic's agentic coding tool, running on datacenter hardware, typically streams output in the ballpark of 50–100 tokens/second, varying with model and load. Now look back at the list: a 3B model on a laptop NPU decodes at 43–57 tok/s. A local model on a battery-powered machine matches the typing speed of a frontier system.
It does not fully match the judgment, though as the Qwen3.8-27B numbers above show, even that gap is now contested territory rather than a chasm. The frontier model still wins on breadth of knowledge and the hardest reasoning; the local model's advantage is that the tokens are free, private, and available on an airplane. The trick to being happy with local AI is the same as staffing anything: match the size of the mind to the size of the task. And what you can run on your own hardware keeps getting bigger faster than I expected it to.
Conclusion
Hopefully this has helped you get a handle on what local AI can and can't do. If you can run it locally, I recommend doing so. Easiest way, by far, is to download Clairvoyance. Once you get comfortable with that, you can branch out. I recommend Ollama and LM Studio.
Now, let's take a step back from the benchmarks for a moment and look at the trajectory. A year ago, running a useful model locally meant enthusiast hardware and a tolerance for pain. Today, a certified laptop you can buy at Best Buy answers in under six seconds, a gaming GPU runs a model that would have been considered frontier a few months ago.
It doesn't take a genius to figure out where things are going. Because make no mistake: There are diminishing returns on the benefit of AI for most people. Just as you don't need an airplane to drive to the store, you don't need a ChatGPT Fable 7 to put together the nightly inventory reports or software crash telemetry. Most use cases of Power BI can be handled now by these Local AI models if you pair them with something like Clairvoyance.
The challenge is going to be integrating these capabilities into an individual or enterprise's workflow stack.