Qwen3.8-27B on the iPhone 18 Neural Engine at 15–70 tok/s (1.5 bpw)
twitter.comOP, asking for upvotes is against the HN guidelines: https://x.com/anemll/status/2111531629529456771
Hi. Could anyone share any recommendations (hopefully with actual links) for App Store apps to try running such local models? I’m not really interested in side-loading since it currently seems (to me) more of an hassle than it’s worth.
App Store search is broken in a sense that there might be so many apps claiming to be local but once you read reviews, it’s just another scam where it’s it’s usually sending your query to remote server.
I'm the dev behind the Private LLM app, which was the first on-device LLM app on the App Store. Don't download it yet. I'm currently finalizing a brand new v2 rewrite which can run models that the copycat app (locally) cannot run, like Bonsai 2 27B, Gemma 4 12B etc.
Thanks! All the best for your rewrite and definitely will check it out once it launches!
My current favorite is this one: https://apps.apple.com/us/app/locally-ai-by-lm-studio/id6741...
It's free, not annoying, and has a bunch of model options.
Non-app-store link: https://locallyai.app/
However this uses the GPU, not the Neural Engine like the submission.
Note that the model discussed in this article isn’t available in this app running on the phone. Still, it has some good local options like Gemma 4 and can link to the desktop LM Studio app which is cool.
Wow! This app is pretty cool!
Not Qwen, Bonsai. Although it is based on Qwen, and achieves similar scores, it is not the "full" Qwen.
Last I remember, when trying to run a model above 1-2 GB on iOS, it would refuse to do so, at least from Locally AI. iOS only allows apps to use about 1-2 GB of RAM before being killed. Though using their Neural APIs might bypass this.
Anyone know how hot an iPhone gets running the Neural Engine flat out for 10 minutes?
It's warm but not hot. On M6 it's about 50% less energy consumption in my tests vs GPU. I think iPhone should be similar
The neural engine is optimized for energy efficiency, and the GPU is optimized for performance.
You're going to get much better battery life and much less heat running on the neural engine.