There are two types of distillation, which is a way to transfer knowledge from a teacher (usually better) model to a student. There are some accusations that Moonshot's Kimi3 was partially distilled from Claude Fable. Even if they did, cutting them off wouldn't halt progress.
Distillation from teacher probabilities
The original definition of distillation by Hinton et al.
Disagreements in student and teacher probability distributions over the next token/word are minimized. This turns out to be a much richer signal than learning from just training on regular text because that puts all probability mass on a single next word, which is wrong (although on average over trillions of tokens it’s sort of OK). A simple illustration is if the next word is ‘big’, it’s silly not to put some mass on ‘large’.
The main downsides are you need access to the tokenizer and the model with its token probabilities, and you need a lot of disk to store those probabilities. Think 256 floats * 10^13 tokens ~ 15 Petabytes for pre-training!
This has been shown to speed up and improve pre-training, and is common practice at labs at least the smaller models. It’s also why the smaller variants (e.g. Terra/Sonnet/Flash) have gotten so good in terms of cost/performance. It is mostly a within-lab technique for training smaller models. API prices would also make it super-expensive for an external customer as a teacher-generated pretraining dataset would cost >$500M from Fable ($50/million tokens).
It’s very unlikely Kimi was distilled in this fashion from US labs as the model APIs no longer return token probabilities and pretraining data is relatively easy to acquire. Instead the Moonshot team should get credit for doing a great job at pretraining. There is no amount of postraining that will fix up a broken pretrained model. This is also historically the most expensive phase in terms of compute, although relatively cheap to get data, since it mainly involves scraping and cleaning up the internet / github / etc, without human labeling.
Distillation from teacher-generated tokens
… the more common practice outside the labs. It is pretty simple: prompt LLM APIs and record the responses, and fine-tune the student on them. Grad students have been doing this since 2022 to post-train base open models.
Why do this? It is primarily to avoid the super-expensive process of acquiring high-quality post-training data. Mercor/Surge/Turing and others have been paid billions by US labs to produce this data, which involves paying increasingly expensive human experts to do things models don’t know how to do already. It is way cheaper to just pay even Fable prices at $50/million tokens. A human paid $10/hr producing 1000 words/hr (which is a lot) would cost roughly $7500/million tokens, i.e. 150x more expensive. Furthermore, you need orders of magnitude less data than pretraining so it is economically feasible to do it in post.
Beyond the cost, it is also much faster and operationally simpler to just call the LLM APIs. No meetings / contracts to sign, setting up a post-training data process, etc.
I don’t think Moonshot has a major technical post-training know-how gap. There’s post-training as a service APIs like Tinker that you can use once you have the data. And you can just sign up as a contractor on the various data collection platforms to see how the data is collected. Engineering-wise it is more difficult to pull off large-scale pre-training, which involves a lot of complex infra as well as acquiring the scare compute.
I don’t know if Moonshot actually distilled Fable/GPT tokens. But even if they did, and they were cut-off, they would still be able to train roughly the same model. It would just take a bit longer, and cost them more money but they’d be able to do it. If anything they may have an advantage in scaling up human expert data collection given China’s massive population of over-educated and lower-paid knowledge workers.
(Only the image in this article was AI-generated)