I ran DeepSeek-V4-Flash on a Raspberry Pi 5 (8GB edition) by streaming model weights from a PCIe attached NVMe SSD. Codex (GPT-5.5 xhigh) and Claude Code (Opus 4.8 max) drove the code and testing, while I provided the steering, architecture, and project management. The whole project took just over a week, with Codex and Claude Code running in goal-mode for most of it, across more than 160 experiments and active testing on the live device.
The end result was DeepSeek-V4-Flash running at 1.3 generated tokens per second with an average power draw of just 8 watts. That is obviously not fast in the way people expect from chatbots or coding agents, but that was never really the point. The point was to see whether a frontier-adjacent open weight model could be made to run fully offline, on cheap junkyard compute, from a device that can be powered by a battery pack.
The GitHub repo for this project is here: danveloper/flash-pi-dsv4
I tweeted about this a few days ago, but I wanted to do a proper write-up of the whole project because this is easily one of the most ambitious and unreasonable technology adventures I’ve undertaken. Running Qwen 3.5 397B on my M3 MacBook was also pretty ambitious, but the unified Apple Silicon architecture with shared memory across CPU and GPU made for a formidable platform. The Raspberry Pi does have a unified memory footprint and GPU, but it is drastically underpowered, with only an ARM Cortex-A76 CPU, a mobile device processor, and in my case, 8GB of low power DDR4. The VideoCore GPU is essentially useless for anything beyond video output and was not used in this project.
There are many reasons for wanting to do this project, but the biggest one is that open weight models are reaching frontier-adjacent levels of capability while still requiring massive compute platforms that are generally unapproachable for the average hobbyist. I wanted to see how far I could push the smallest, cheapest, off-the-shelf compute against one of the best open weight models available, trading speed for lower cost and lower power requirements.
Huge GPU clusters are unaffordable and require tremendous amounts of power. Hyperscalers (Google, Amazon, Microsoft, …) and the Big Labs (Anthropic, OpenAI, Google, xAI, …) are actively buying up as much power, and as many GPUs and RAM chips as they can, in order to serve a rapidly growing economy of AI tokens. This leaves the future of AI and technology concentrated in the hands of a few companies with the capital, procurement pipelines, energy contracts, and datacenter access to acquire as much compute and power as they can. And while an open weight model from a lab on the other side of the world will never compete on the actual frontier of AI from US labs investing half a trillion USD a year, it may be Good Enough for a hobbyist home lab.
Most important to the goal of my project was demonstrating AI inference on the lowest power footprint possible, which made the Raspberry Pi an appealing target platform. Applications for this level of intelligence, like robotics or advanced automation, will need to be considerate of continual power draw. My supposition is that unattended automation doesn’t need the same real-time throughput as a chat interface or coding agent, but will most certainly benefit from a higher intelligence model. If you’re not sitting there watching the tokens stream, it may be acceptable to have a platform that can reason at 1-2 tokens per second and take actions over the course of minutes and hours.
For a rough power-envelope comparison, a $50 USD battery pack can provide up to 74Wh (watt-hours) of power, which would give about 9 hours of full-tilt inference at ~1tok/s before needing recharge, or approximately 32,000 tokens of generation. Now, this is obviously never going to compete with an API call to an online AI provider, but the idea of having a fully offline, highly customized, highly intelligent agent running on “junkyard compute” hopefully inspires some ideas for what’s possible.
How Does it Work?
In short, the trick of this project is that only the parts of the model that actively need to be computed are loaded in memory at any point in time. The rest stays on the SSD and is made active just-in-time. This allows a gigantic model that would normally require a datacenter-class memory footprint to run on a tiny platform with 8GB of RAM. There are a thousand micro-optimizations underneath that, but the core idea is simple in that all we need to do is keep the CPU busy, keep the SSD reading, and avoid loading bytes that are not about to be used.
The project uses Antirez’s “DwarfStar4” GGUF model, which is a 4bit quantization of the shared model weights and a clever iMatrix 2bit quantization of the routed experts. That matters because DeepSeek-V4-Flash is a mixture-of-experts model. In an MoE transformer, tokens still move through the model’s layer stack, but some feed-forward blocks contain multiple expert MLPs. A router chooses which experts should process each token at that layer. In Antirez’s model, those routed experts are quantized down to 2bit, which significantly reduces how much data needs to be loaded for each active expert.
The default configuration routes each token through a certain number of experts on each forward pass. In DeepSeek-V4-Flash’s case, that’s 6 experts. You can reduce the number of active experts by taking a quality hit, and in my case I found that 4 experts was a reasonable cost-to-quality tradeoff, though the difference is not entirely unnoticeable. The biggest risk of reducing it further is that the model goes smooth-brained and starts repeating itself or outputting garbage results. In the case of Qwen, I binary searched the number to figure out when it started outputting JSON strings properly. In the case of DeepSeek, I reduced it down to 2 and found conversation and code quality was still acceptable, so the final configuration for my project only uses 2 experts per forward pass. YMMV.
Weights stream from disk into RAM as needed, and this works mostly because NVMe SSD throughput has gotten absurdly good. It is not a replacement for RAM in the general case, and I do not want to pretend SSDs are secretly DDR4, but for carefully ordered sequential reads, modern NVMe storage is fast enough to become interesting as a streaming substrate. The Pi 5’s PCIe 3.0 path gave us a theoretical max of about 985 MB/s. In practice, on ext4 with 64kb readahead sizes, the engine could pull about 600MB/s of sustained data. The final calculus was not “make disk fast” in the abstract. It was “make disk fast enough, with the right bytes, to keep the CPU saturated.”
That last part is really important. After the model makes an expert routing decision, the engine needs to seek to the expert’s location inside the weight payload, grab the bytes for that expert, load them into memory, and keep the inference pipeline moving. The enemy is idle time. You do not want the CPU sitting around waiting on disk, and you do not want the disk eagerly loading a bunch of irrelevant weights that the CPU will never touch.
By design, experts in GGUF are packed by tensor type, for example “gate”, “up”, and “down”. For our purposes, that means the tensors for an individual expert are spaced across hundreds of megabytes of weights that are not relevant to the current computation. In the native tensor-major format, reading one expert means doing many smaller reads in sequence, jumping from the expert’s gate tensor, to its up tensor, to its down tensor, with a bunch of unrelated bytes in between.
That is no good for high-throughput streaming inference, so we repacked the experts into expert-major format. Instead of storing all gates together, all ups together, and all downs together, we stored each expert’s active bytes contiguously on disk. When the router selects an expert, the engine can issue one large read for the bytes it actually needs. Aligning those chunks to 2MB strides gave us high read throughput without accidentally dragging irrelevant weights into RAM. In short, we “defragged the weights”.
The model runs entirely on the Pi’s CPU, which is normally a horrendously underpowered device for this kind of task. Nonetheless, I was astounded at how capable the Cortex-A76 actually is. It’s a quad-core ARM chip with NEON SIMD support, which gives us a path for low-precision matrix multiplication. A custom NEON kernel fuses the relevant matrix math operations so the CPU can do more useful work per invocation, with less overhead and better cache behavior.
The aligned weights and fused kernel are what allow the CPU and SSD to stay busy at the same time. While the CPU is computing the current expert, the engine can prepare the next read. In practice, there is still a little headroom on both CPU and disk. I measured average sustained CPU usage around 87% and disk throughput around 600MB/s compared to the theoretical PCIe ceiling of 985MB/s. There may be further optimizations that get closer to the theoretical limit, but I was not able to find them here.
Quickly going back to the CPU’s SIMD support: the feed-forward portion of a transformer layer is a set of sequential computations, including the gate, up, and down projections from earlier. Each of these computations uses scratch space where intermediate activations are briefly stored for the next calculation. Typically, those activations are stored in 16- or 32-bit precision. In this project, we keep the scratch space at 8bit precision to improve SIMD efficiency and keep the memory footprint low. When you read a model configuration like “W8A16”, it means the weights are stored or loaded at 8bit precision and the activations are held at 16bit precision in scratch space. Here, the goal was to keep both the streamed weights and the temporary activations as cheap as possible for the Pi’s CPU and memory system.
Where This Approach Hurts Most...
The metric everybody focuses on, and I have also intentionally focused on, is tokens per second. That is the throughput of generation speed: the “Good” and “morning” and “!” that you see appearing on the screen. But inference actually breaks down into two primary phases: prompt processing and generation, usually called prefill and decoding.
Before prefill, the user’s input is tokenized, which means the input string is split into predefined chunks and each chunk is assigned a number. Prefill then processes all of those prompt tokens through the transformer layers so each attention layer can compute and store the prompt tokens’ key and value states in the KV cache. During generation, each new token computes its own query and uses it to attend against those cached keys and values.
Normally, prefill can be parallelized across the prompt because the full input sequence is known up front. The attention mask preserves causal ordering, and the result is a populated KV cache for generation. That parallelism is why a big GPU can ingest a long prompt quickly before the first generated token appears.
In this project, prefill is hit hardest. Since the Pi is streaming weights from disk and doing all compute on the CPU, prompt processing does not get the huge parallel speedup people expect from normal inference hardware. Long prompts take a long time to compute before the first generated token appears. That makes this setup unreasonable for cold, long, dynamic context processing, and therefore not great for chatbot-style interaction.
That said, it is not a dealbreaker for every use case. For a fixed instruction, a long KV cache could be computed on a faster machine and reified at inference time, essentially nullifying the prefill cost. That is not great for a chatbot, but it could be reasonable for a model following the same instruction at every invocation, for example, checking a sensor or metric and deciding whether to take some action.
This is the bigger point: unattended automation does not necessarily need the same latency profile as a chat window. If the agent is not being watched token by token, a slow but capable offline model may still be useful, especially if the task unfolds over minutes or hours instead of milliseconds.
What’s Next?
It’s actually funny to me that this project ended up where it did, because I had originally set out to build a prefill accelerator using the Raspberry Pi + Hailo 10H HAT. The Hailo chip is exciting to me because it lets you compile a model into the physical execution layout of the accelerator and run inference at a few watts. I spent the arduous dozens of hours converting the DeepSeek weights into the HEF format the Hailo needs, and I was able to validate 2 layers fully. It should have occurred to me sooner, but the Hailo cannot do dynamic routing in the way this model requires, so MoE routing is completely off the table for that device.
That failure ended up being clarifying... This project demonstrates a viable path for low-cost inference on cheap hardware, but it also demonstrates how hostile current MoE architectures are to streaming inference. MoE models are great for training and efficient when the weights are already resident in memory, but for this project it is brutal to not know the next expert until the current layer has done its routing work. When all model layers are in memory, that is just a routing decision. When model weights are being streamed from disk, that routing decision determines what bytes need to exist in RAM next.
For streaming model weights, it would be far better if expert layers were linearly aligned downstream. Meaning, if the engine could know which experts layer N+1, N+2, N+n would need, it could eagerly prepare those reads while layer N was computing, and both the CPU and disk could operate closer to their maximum potential. We would probably see another 2-3 tokens per second on the Raspberry Pi for a model architecture designed around that constraint.
Importantly, a linearly aligned MoE would also reopen the door for accelerators like the Hailo. The dynamic decision-making could stay on the CPU, while the predictable parts of inference could move to the accelerator. The exact TOPS comparisons across chips are messy because vendors quote different precision modes and different accelerators, but the important point is power density. The Hailo is built to deliver a large amount of low-precision inference throughput at only a few watts. If more of the model’s execution path were statically knowable, prefill acceleration on this kind of device would become much more realistic.
Besides that, I think there’s a bigger opportunity for small models to demonstrate much higher capability. It seems like as the months go by, researchers are packing more intelligence into smaller architectures. OpenAI’s parameter-golf contest recently challenged model makers to get to GPT-2, a 124M parameter model, quality in a model architecture that fits in an under 16MB weight artifact and can be trained in under 10 minutes. The leaderboard is basically recursive models, where layers are looped and weights are packed tightly. A model with a few million parameters today can compare surprisingly well to much larger models from only a few years ago. That is incredible progress, and I am confident small recursive models are a future path for high-throughput local inference.
It was never the point of this project to make the Raspberry Pi competitive with a GPU server. The point is that with quantization, streaming, layout-aware weights, and a willingness to trade latency for autonomy, frontier-adjacent intelligence starts to become something you can run from a battery pack on commodity hardware. Progress is not slowing on AI, not by a long shot, and I intend to continue pushing the bounds of what is possible on commodity, off-the-shelf consumer hardware. Big and small models can and will run everywhere.
The content of this post was written by me and recomposed for flow and readability by GPT-5.5.