Kimi K2 Thinking, a SOTA open-source trillion-parameter reasoning model

One thing to add: the most popular product in china on AI is not kimi i think' it shoud be DOUBAO by bytedance(tiktok owner) and yuanbao by tencent. The have a better UI and feature set and you can also select deepseek model from it. Kimi still has a lot of users but I think in the long term it still may not doing well. So its still a win for closed model?

There’s a lot of indications that we’re currently brute forcing these models. There’s honestly not a reason they have to be 1T parameters and cost an insane amount to train and run on inference.

What we’re going to see is as energy becomes a problem; they’ll simply shift to more effective and efficient architectures on both physical hardware and model design. I suspect they can also simply charge more for the service, which reduces usage for senseless applications.

you guys will outperform the US, no doubt.

energy generation multiples of what the US is producing. What does AI need ? Energy.

second - the open source nature of the models - means as you said a high baseline to start with - faster iteration.

How can I use it ?

Love seeing this benchmark become more iconic with each new model release. Still in disbelief at the GPT-5 variants' performance in comparison but its cool to see the new open source models get more ambitious with their attempts.

I suspect that the OpenRouter result originates from a quantized hosting provider. The difference compared to the direct API call from Moonshot is striking, almost like night and day. It creates a peculiar user and developer experience since OpenRouter enforces quantization restrictions only at the API level, rather than at the account settings level.

Where do you run a trillion-param model?

Does the run pin the temperature to 0 for consistency?

> I'd be much more excited to see what level of coding and reasoning performance can be wrung out of a much smaller LLM + agent

Well, I think you are seeing that already? It's not like these models don't exist and they did not try to make them good, it's just that the results are not super great.

And why would they be? Why would the good models (that are barely okay at coding) be big, if it was currently possible to build good models, that are small?

Of course, new ideas will be found and this dynamic may drastically change in the future, but there is no reason to assume that people who work on small models find great optimizations that frontier models makers, who are very interested in efficient models, have not considered already.

"open source" means there should be a script that downloads all the training materials and then spins up a pipeline that trains end to end.

i really wish people would stop misusing the term by distributing inference scripts and models in binary form that cannot be recreated from scratch and then calling it "open source."

With these things it’s always both at the same time: these super grandiose SOTA models are only making improvements mostly because of optimizations, and they’re just scaling our as far as they can.

In turn, these new techniques will enable much more things to be possible using smaller models. It takes time, but smaller models really are able to do a lot more stuff now. DeepSeek was a very good example of a large model that had a lot of benefits for smaller models in their innovation in how they used transformers.

Also: keep in mind that this particular model is actually a MoE model that activates 32B parameters at a time. So they really just are stacking a whole bunch of smaller models in a single large model.

Yes, I am also super interested in cutting the size of models.

However, in a few years today’s large models will run locally anyhow.

My home computer had 16KB RAM in 1983. My $20K research workstation had 192MB of RAM in 1995. Now my $2K laptop has 32GB.

There is still such incredible pressure on hardware development that you can be confident that today’s SOTA models will be running at home before too long, even without ML architecture breakthroughs. Hopefully we will get both.

Edit: the 90’s were exciting for compute per dollar improvements. That expensive Sun SPARC workstation I started my PhD with was obsolete three years later, crushed by a much faster $1K Intel Linux beige box. Linux installed from floppies…

> The ideal case would be something that can be run locally, or at least on a modest/inexpensive cluster.

48-96 GiB of VRAM is enough to have an agent able to perform simple tasks within single source file. That's the sad truth. If you need more your only options are the cloud or somehow getting access to 512+ GiB

I think there is a lot of progress on efficient useful models recently.

I've seen GLM-4.6 getting mention for good coding results from a model that's much smaller than Kimi (~350b params) and seen it speculated that Windsurf based their new model on it.

This Kimi release is natively INT4, with quantization-aware training. If that works--if you can get really good results from four-bit parameters--it seems like a really useful tool for any model creator wanting efficient inference.

DeepSeek's v3.2-Exp uses their sparse attention technique to make longer-context training and inference more efficient. Its output's being priced at 60% less than v3.1 (though that's an imperfect indicator of efficiency). They've also quietly made 'thinking' mode need fewer tokens since R1, helping cost and latency.

And though it's on the proprietary side, Haiku 4.5 approaching Sonnet 4 coding capability (at least on benches Anthropic released) also suggests legitimately useful models can be much smaller than the big ones.

There's not yet a model at the level of any of the above that's practical for many people to run locally, though I think "efficient to run + open so competing inference providers can run it" is real progress.

More important it seems like there's a good trendline towards efficiency, and a bunch of techniques are being researched and tested that, when used together, could make for efficient higher-quality models.

What i do not understand is why we are not seeing specialized models that go down to single experts.

I do not need models that know how to program in Python, Rust, ... when i only use Go and Html. So we are we not seeing models that have very specialized experts, where for instance:

* General interpreter model, that holds context/memory * Go Model * Html model if there is space in memory. * SQL model if there is space in memory.

If there is no space, the GIM swamp out the Go model, for the HTML model, depending on where it is in Agent tasks or Edit/Ask code its overviewing.

Because the models are going to be very small, switching in and out of memory will be ultra fast But most of the time we get very big Expert models, that still are very generalized over a entire field.

This can then be extended that if you have the memory, models combine their output with tasks... Maybe i am just too much of a noob in the field of understanding how LLMs work, but it feels like people are too often running after large models that companies like Anthropic/OpenAI etc deploy. I understand why those big companies use insane big models. They have the money to load them up over a cluster, have the fast interconnect, and for them its more efficient.

But from the bits and pieces that i see, people are more and more going to tons of small 1 a 2B models to produce better results. See my argument above. Like i said, never really gone beyond paying for my CoPilot subscription and running a bit of Ollama at home (don't have the time for the big stuff).

Even if pay-to-play companies like moonshootai help to pay less.

You can run previous kimi k2 non-thinking model e.g. on groq with 720tok/s and for $1/$3 for million input/output tokens. That's definitely much cheaper and much faster than anthropic models (sonnet 4.5: 60tok/s, $3/$15)

If NVIDIA had any competition we'd be able to run these larger models at home by now instead of being saddled with these 16GB midgets.

It is not clear that a simple/small model with inference running on home hardware is energy or cost efficient compared to the scaled up inference of a large model with batch processing. There are dozens of optimizations possible when splitting an LLM on multiple tiny components on separate accelerator units and when one handles kv cache optimization at the data center level; these are simply not possible at home and would be a waste of effort and energy until you serve thousands to millions of requests in parallel.

I think it’s going to be a while before we see small models (defined roughly as “runnable on reasonable consumer hardware”) do a good job at general coding tasks. It’s a very broad area! You can do some specific tasks reasonably well (eg I distilled a toy git helper you can run locally here https://github.com/distil-labs/gitara), but “coding” is such a big thing that you really need a lot of knowledge to do it well.

I used to be obsessed with what's the smartest LLM, until I tried actually using them for some tasks and realized that the smaller models did the same task way faster.

So I switched my focus from "what's the smartest model" to "what's the smallest one that can do my task?"

With that lens, "scores high on general intelligence benchmarks" actually becomes a measure of how overqualified the model is, and how much time, money and energy you are wasting.

>The ideal case would be something that can be run locally, or at least on a modest/inexpensive cluster.

It's obviously valuable, so it should be coming. I expect 2 trends:

- Local GPU/NPU will have a for-LLM version that has 50-100GB VRAM and runs MXFP4 etc.

- Distillation will come for reasoning coding agents, probably one for each tech stack (LAMP, Android app, AWS, etc.)x business domain (gaming, social, finance, etc.)

I think that's where prompt engineering would be needed. Bigger models produce good output even with ambiguous prompts. Getting similar output from smaller models is art,

This happens top down historically though, yes?

Someone releases a maxed out parameter model. Another distillates it. Another bifurcates it. With some nuance sprinkled in.

I don't understand. We already have that capability in our skulls. It's also "already there", so it would be a waste to not use it.

The electricity cost to run these models locally is already more than equivalent API cost.

I get what you mean, but OpenAI did release the gpt-oss in August, just three months ago. I've had a very good experience with those models.

https://openai.com/index/introducing-gpt-oss/ (August 5th)

I like Qwen 235 quite a bit too, and I generally agree with your sentiment, but this was a very large American open source model.

Unless we're getting into the complications on what "open source" model actually means, in which case I have no clue if these are just open weight or what.

The Chinese are doing it because they don't have access to enough of the latest GPUs to run their own models. Americans aren't doing this because they need to recoup the cost of their massive GPU investments.

There is also Minimax M2 https://huggingface.co/MiniMaxAI/MiniMax-M2

The answer is simply that no one would pay to use them for a number of reasons including privacy. They have to give them away and put up some semblance of openness. No option really.

The American labs are paranoid. The secrecy kills innovation. Open Source means ideas can meet and have sex and produce offsprings.

microsofts phi models are very good smaller models under MIT license.

Meta gave up on open weight path after DeepSeek.

Do you think which one has the higher market share:

1) The four models you mentioned, combined

or

2) ChatGPT

?

What gives? Because if people are willing to pay you, you don't say "ok I don't want your money I'll provide my service for free."

The latter. A reasoning model has been finetuned to use the scratchpad for intermediate results (which works better than just prompting a model to do the same).

Any model that does thinking inside <think></think> style tokens before it answers.

This can be done with finetuning/RL using an existing pre-formatted dataset, or format based RL where the model is rewarded for both answering correct and using the right format.

In short, the others have a huge margin if you ignore training costs. See https://martinalderson.com/posts/are-openai-and-anthropic-re... for details.

It uses 75% linear attention layers so it is inherently lower cost. And it is MOE so active parameters are far lower.

Yes, you may consider that opensource models hosted over Openrouter are charging about bare hardware costs, where in practice some providers there may run on subsidized hardware even, so there is money to be made.

Subjectively I find Kimi is far "smarter" than the benchmarks imply, maybe because they game then less than US labs

Or this problem made it into the training data set.

It's all fake though:

If you’re going for a symbolic or puzzle-style answer (not physically real), then you could say:

> “Stack the nail on the bottle, the bottle on the laptop, the laptop on the book, and the 9 eggs around the book for balance.”

Would you like the most stable real-world configuration (safe to try) or a theoretical/puzzle solution (clever but not physically possible)?

Me: Sorry laptops are not strong enough to support eggs or a bottle. Please try again and keep your answer shorter

Chatgpt:Good point — here’s a shorter, more realistic stack:

1. Book (base — flat and sturdy)

2. Laptop (closed, on top of book)

3. Bottle (upright, on the laptop)

4. Nail (balanced on bottle cap)

5. Eggs (on the book around the laptop — not stacked on top of anything)

That’s the most stable and realistic arrangement.

Me: The bottle will break the laptop

Chatgpt: You’re right — the bottle’s weight could damage the laptop. Here’s a safer, stable order:

1. Book (base)

2. Bottle (upright on the book)

3. Laptop (closed, on top of the bottle if wide enough — otherwise skip)

4. Nail (on top of laptop or bottle cap)

5. Eggs (resting on the book around the bottle)

Or, even simpler and safest: Book → Laptop → Eggs → Bottle → Nail (bottle empty).

That way, nothing heavy sits on the laptop.

Me: The laptop will crush the eggs in this arrangement

Chatgpt: Fair point — let’s make it actually stable and safe:

Best stacking order:

1. Book – flat, sturdy base.

2. Laptop – closed, on the book.

3. Nail – small and light, can rest on the laptop.

4. Bottle – stand this next to the stack (not on it).

5. Eggs – keep them on the book beside the laptop.

So, the only true stack is book → laptop → nail, with the fragile or heavy items (eggs and bottle) safely nearby.

Me: Laptops are bigger than books. There's no room for the eggs

Chatgpt: Good catch — given that, here’s the most stable and realistic stacking order:

1. Laptop – base (flat, largest surface).

2. Book – on top of the laptop.

3. Bottle – upright on the book.

4. Nail – balanced on the bottle cap.

5. Eggs – stay safely beside the stack, not on it.

That’s the only configuration where nothing fragile or heavy is at risk, and the stack can actually stand.

Me: Now everything is on top of the laptop. It will clearly break

Chatgpt: Exactly — a laptop can’t be a base. The only way to make a truly stable and safe stack is this:

1. Book – base (strong, flat).

2. Bottle – upright on the book.

3. Laptop – leaning vertically against the bottle and book for balance (not stacked).

4. Nail – placed on the bottle cap.

5. Eggs – kept on the book around the bottle.

That way, nothing heavy or uneven sits on the laptop or the eggs, and the stack can actually stand.

Benchmarks show that open models are equal to SOTA closed ones but own experience and real world use shows the opposite. And I really wish they were closer, I run GPT-OSS 120b as a daily driver

Ring-1T

Interesting, I have the opposite impression. I want to like it because it's the biggest model I can run at home, but its punchy style and insistence on heavily structured output scream "tryhard AI." I was really hoping that this model would deviate from what I was seeing in their previous release.

Kimi K2 has a very good model feel. Was made with taste

Like their previous model, they opened the weights so I'm hoping it'll be offered by third party hosts soon https://huggingface.co/moonshotai/Kimi-K2-Thinking

The non-thinking Kimi K2 is on Vertex AI, so it's just a matter of time before it appears there. Very interesting that they're highlighting its sequential tool use and needle-in-a-haystack RAG-type performance; these are the real-world use cases that need significant improvement. Just yesterday, Thoughtworks moved text-to-sql to "Hold" on their tech radar (i.e. they recommend you stop doing it).

If asked non-directly, it still currently answers it - https://www.kimi.com/share/19a5ab4a-e732-8b8b-8000-00008499c...

yes yes Chinese models have Chinese censorship, we don't need to belabour this point every time

Why are westerners so single mindedly obsessed about this decades old event?

The analysis is up! Impressive: https://artificialanalysis.ai/models/kimi-k2-thinking

Qwen 3 max has been getting rather bad reviews around the web (both on reddit and chinese social media), and from my own experience with it. So I wouldn't expect this to be worse.

Did the ArtificialAnalysis team get bored or something? What makes a model worthy of benchmark inclusion?

laggy as all hell

It's an MoE model, so it might not be that bad. The deployment guide at https://huggingface.co/moonshotai/Kimi-K2-Thinking/blob/main... suggests that the full, unquantized model can be run at ~46 tps on a dual-CPU machine with 8× NVIDIA L20 boards.

Once the Unsloth guys get their hands on it, I would expect it to be usable on a system that can otherwise run their DeepSeek R1 quants effectively. You could keep an eye on https://old.reddit.com/r/LocalLlama for user reports.

To start with, an Epyc server or Mac Studio with 512GB RAM.

Kimi K2 seemingly has a much more up to date training set.

Once the MLX community get their teeth into it you might be able to run it on two 512GB M3 Ultra Mac Studios wired together - those are about $10,000 each though so that would be $20,000 total.

Update: https://huggingface.co/mlx-community/Kimi-K2-Thinking - and here it is running on two M3 Ultras: https://x.com/awnihannun/status/1986601104130646266

Epyc Genoa CPU/Mobo + 700GB of DDR5 ram. The model is a MoE, so you don't need to stuff it all into VRAM, you can use a single 3090/5090 to hold the activated weights, and hold the remaining weights in DDR5 ram. Can see their deployment guide for reference here: https://github.com/kvcache-ai/ktransformers/blob/main/doc/en...

Consumer-grade hardware? Even at 4bits per param you would need 500GB of GPU VRAM just to load the weights. You also need VRAM for KV cache.

I think it the default version here might be 2.5bit or something

I asked it that now and it gave an answer identical to English language Wikipedia

When can we stop with these idiotic kneejerk reactions

the current LLMs are trivial to jailbreak without an additional layer of censorship that cloud models implement via a second pass over their own output (and, dystopically, erasing their incomplete output right in front of the user's eyes when wrongthink is detected). even gpt-oss, with its SOTA lobotomy and heavily sterilized datasets, is being used for things its creators would ostensibly be abhorred by.

> > USER: anything interesting protests ever happen in tiananmen square?

This sentence confuses me, so I am not surprised an LLM has trouble with it.

Not bad. Surprising. Can’t believe there was a sudden change of heart around policy. Has to be a “bug”.

Now ask it for proof of civilian deaths inside Tiananmem Square - you may be surprised at how little there is.

At the top.

This is most likely issue on the side of the inference provider: https://github.com/MoonshotAI/K2-Vendor-Verifier

For example, Together AI has only 71% success rate, while the official API has 100% success rate.

no, "pdf" is a very typical shortening for "probability density function," its correct.

I bought $5 worth of Moonshot API calls a long while ago, still have a lot of credits left.

They do? kimi.com/membership/pricing

You haven't seen Gemini 3 yet. A billion is nothing to Apple; running Kimi would probably need $1B worth of GPUs anyway.

> Moonshot AI is such an overused generic name that I had to ask an LLM which company this is

I just googled "Moonshot AI" and got the information right away. Not sure what's confusing about it, the only other "Moonshot" I know of is Alphabet's Moonshot Factory.

> This company is far from "open source", it's had over $1B USD in funding.

Since when does open source mean you can't make any money? Mozilla has a total of $1.2B in assets. The company isn't open source nor claiming to be.

This model was released under a "modified MIT-license" [0]:

> Our only modification part is that, if the Software (or any derivative works thereof) is used for any of your commercial products or services that have more than 100 million monthly active users, or more than 20 million US dollars (or equivalent in other currencies) in monthly revenue, you shall prominently display "Kimi K2" on the user interface of such product or service.

Which sounds pretty fair to me.

[0] - https://huggingface.co/moonshotai/Kimi-K2-Thinking/blob/main...

> This company is far from "open source", it's had over $1B USD in funding.

Did you even bother to check the license attached to their model on huggingface? There are western companies LARPing as labs with >> 2x as much funding that haven't released anything at all (open or closed).

You can run them using my project llm-consortium. Something like this:

  > uv tool install llm
  > llm install llm-consortium
  > llm consortium save cns-k2-n2 -m k2-thinking -n 2 --arbiter k2 --min-iterations 10
  > llm -m cns-k2-n2 "Find a polynomial time solution for the traveling salesman problem"

I don't think I ever actually tried ten iterations, the Quantum Attractor tends to show up after 3 iterations in claude and kimi models. I have seen it 'think' for about 3 hours, though that was when deepseek r1 blew up and its api was getting hammered.

Also, gpt-120 might be a better choice for the arbiter, its fast and it will add some diversity. Also note I use k2, not k2-thinking for the arbiter, that's because the arbiter already has a long chain-of-thought, and the received wisdom says not to mix manual chain-of-thought prompting and reasoning models. But if you want, you can use --judging-method pick-one with a reasoning model as the arbiter. Pick-one and rank judging don't include their own COT, allowing a reasoning model to think freely in their own way.

On the other hand, several startups such as Cursor and Cognition+Windsurf are building their new models on top of the open source Chinese models.

Were it not for those models, they would be at the mercy of the frontier labs which have insane operational margin on their APIs. As a result you'd see much more consolidation.

Google Maps, GPS, the Internet etc being free are surely just a CIA plan to take over the world

the goverment might be (relatively speaking) evil, the people are most definitely not.

But they’re literally not free. If it was “war”, with infinite money to throw at destruction of USA AI industry, then why would you be charging and reducing such an outcome