This technical report (https://arxiv.org/abs/2501.12599) presents our results, and I believe everyone would like to know about the thought process. So I'd like to share some key thinking processes in replicating o1, which is my own Long Chain of Thoughts.
On September 12, 2024, o1 was released, and it was stunning with explosive results. The effectiveness of Long CoT led me to deep reflection.
The effectiveness of Long CoT was actually known more than a year ago. Tim (co-founder of Moonshot AI) verified very early that using small models to train models for multi-digit arithmetic operations, synthesizing detailed calculation processes into long CoT data for SFT, could achieve excellent results. I still remember how shocked I was seeing those results. While we recognized the importance of Long Context and prioritized extending context length, we didn't pay enough attention to Long CoT. This was mainly due to cost considerations. Long Context primarily deals with long text input, with Prefill and Mooncake support, making costs and speed manageable. However, Long CoT involves long text output, which is much more costly and slower. Under these circumstances, extending output length wasn't considered a high priority.
But what's more important than Performance? Costs and speed benefit from Moore's Law and can continuously decrease. Once performance is improved, everything else becomes secondary.
Therefore, we needed to work on Long CoT, to work on o1.
But how exactly should we approach this?
We needed to gather information first to reverse engineer how o1 and RL-LLM might work (yes, this triggered my search API).
First, observing examples from o1's official website, we noticed some distinct characteristics:
o1 can make mistakes!!! Long CoT is different from previous CoT.
o1 often repeatedly reflects and tries again, with various "but, wait,..."
o1's thinking methods are unrestricted, capable of restating problems, making associations, using divide-and-conquer approaches...
Then, two important OpenAI videos came out, from Noam Brown and Hyung Won Chung:
https://www.youtube.com/watch?v=eaAonE58sLU
https://www.youtube.com/watch?v=kYWUEV_e2ss
I watched these videos multiple times because although they weren't recorded recently, they were only released after o1's launch, indicating a close connection between their talks and o1's implementation.
Among them, I found two slides particularly crucial:
This one from Noam Brown highlighted the importance of Test-Time Search, which reminds me of how AlphaGo worked. Many people thought Noam Brown was suggesting implementing AlphaGo's MCTS in LLMs, but that's not the case - he was simply emphasizing the importance of search. When applied to LLMs, it means:
We need to enable models to search on their own!
This led me to revisit Richard Sutton's The Bitter Lesson:
http://cs.utexas.edu/~eunsol/courses/data/bitter_lesson.pdf
Sutton had revealed everything long ago.
Then came Hyung Won Chung's "Don't Teach, Incentivize," which provided even greater insights:
Why did Hyung Won Chung particularly emphasize this? Structure, what is Structure?
MCTS is a structure, A* is a structure. When we artificially add inductive bias and force LLMs to think in our prescribed format, that thinking becomes structured. Based on our earlier observations of o1, we can connect these two points:
o1 doesn't restrict how the model thinks!
This point is extremely crucial! Therefore, we can already rule out some structure-based approaches - no need to explore MCTS because it definitely won't work.
Specifically:
Current various Agentic Workflows are all structure-based things, which will inevitably limit the model's capabilities.
Thus, we can directly conclude:
Agentic Workflow only has short-term value, not long-term value! It will eventually be replaced by the model's inherent capabilities.
All in all, we need to train models to think like humans - to think freely!
Then, there's another important slide from Noam Brown:
This directly tells us:
Do RL with exact rewards! Don't be constrained by Reward Models.
Those who have worked with RLHF know this is a very tricky issue, as Human Preference cannot be modeled precisely, and trained Reward Models are prone to reward hacking. Lilian Weng even wrote a specific blog about this:
Reward Hacking in Reinforcement Learning https://lilianweng.github.io/posts/2024-11-28-reward-hacking/
So, if rewards aren't accurate, RL becomes difficult to work effectively. RL performance completely depends on rewards.
This reminds me of the extensive reward shaping we did in robot locomotion and game AI.
The message becomes very clear:
Practice problems, practice problems, and more practice problems! Work on problems with standard answers!
Math and code are two straightforward directions with standard answers. OpenAI is focusing on these, so we should too.
Worried about forgetting due to long context, let's confirm the key information we've gathered:
Train LLMs through RL to solve problems with exact rewards
Avoid structured approaches as they ultimately limit model performance; let the model explore thinking paradigms on its own
Thinking includes search processes and allows for mistakes
Now, we need to solve how to train LLMs using RL through Long CoT to solve problems.
Many might think: Just use PPO!
But it's not that simple. Long CoT significantly changes how we model the problem.
o1 is actually In Context RL with Self-Critique
o1 actually trains the complete trajectory of in-context RL as a single message.
As shown in the following code:
What is In Context RL?
In-context Reinforcement Learning with Algorithm Distillation arxiv.org/abs/2210.14215
For those unfamiliar, simply put, the model's next token prediction process itself is an RL exploration process.
When a model solves a problem under Long CoT, what is it actually doing?
It's actually learning to solve the problem, and its entire trajectory is:
s1,a1,r1,a2,r2,a3,r3,.....
Here, 'a' can be considered a solution method, viewed as an action, while 'r' is the reward, but in this case, it's a reward obtained through the model's own reflection, which is why I call it in-context RL with built-in critic/world model.
A recent related blog also finally addressed this topic, worth checking out:
Optimizing LLM Test-Time Compute Involves Solving a Meta-RL Problem https://blog.ml.cmu.edu/2025/01/08/optimizing-llm-test-time-compute-involves-solving-a-meta-rl-problem/
However, it doesn't address the self-critique aspect.
So this matter becomes more complex.
If we model long CoT output as in-context RL with self-critique, how do we optimize it?
The primary challenge is determining the value of each statement.
You'll find this value becomes extremely difficult to estimate.
Take a very simple example: Calculate 1+1=?
Then the model outputs: "1+1=3, no that's wrong, is it 1+1=4? That's wrong too, because 4-1=3. What about 1+1=1? No, because 1 doesn't equal 1-1=0... Oh, I get it - if I have one candy in my left hand and one candy in my right hand, and I put the left hand candy in my right hand, then I have two candies in my right hand. Now I understand, 1+1=2"
You'll discover:
If the model doesn't reflect, making a mistake would be wrong, and the value would be negative. But if the model can reflect, as long as it recognizes and corrects mistakes to reach the right answer, these errors shouldn't count as mistakes. The value shouldn't be negative.
From this, we can see that whether a Policy can self-reflect leads to vastly different values. We cannot use an off-policy value to optimize Policy, and in next token prediction scenarios, it's extremely difficult to decompose steps, and it's nearly impossible to perform large-scale MC rollouts at each step to estimate current value. Furthermore, imagine a scenario with an almost infinite context length (10M), like a game of Go with almost unlimited move undos - this scenario approaches a non-MDP state, making value estimation extremely challenging.
This reminds me of life:
o1 is like life itself. Life is a finite sequential trajectory, full of exploration and mistakes. Except for serious crimes, you can't evaluate whether certain mistakes/setbacks are good or bad (like Steve Jobs being fired from the company he founded) - the final outcome depends on your own goals.
Therefore, we can logically conclude that we shouldn't train value functions or develop PRMs, as they won't be accurate.
So, the problem of training LLMs through RL with Long CoT seems simpler now:
Regardless of what mistakes the model makes in the process, as long as they're not repetitive and the model eventually gets it right, we consider it good exploration worth encouraging. Conversely, if the model explores but ultimately gets it wrong, no matter how hard it tried, it deserves punishment.
We've transformed this into a Contextual Bandit problem. So we can use a REINFORCE variant to solve it.
Above is the basic REINFORCE formula - simply put, add gradient for correct answers, subtract for wrong ones. Of course, we need to make training more stable by adding KL divergence, reward normalization, and other tricks, which you can find in the paper (https://arxiv.org/abs/2501.12599). But the basic idea is this simple.
With the basic approach established, what remains is implementation. However, there's still one important question:
How does Long CoT grow longer? This might be the most crucial aspect.
Surprisingly, during our actual training, we made an important discovery:
As models improve performance through training, they naturally increase token count!
This emerges spontaneously during the RL training process!
Fantastic!
This discovery is almost identical to what Deepseek found. Their zero-sft RL approach is quite impressive!
OK, clarifying this whole thought process wasn't easy. I feel like an AI - no, rather, AI has become very human-like.
Our thinking continues, so what about the future?
Well, o3 is ahead, and there's still much to explore.
AGI is indeed just around the corner, Sam Altman wasn't exaggerating about that.
What about ASI (Artificial Super Intelligence)?
I've started reconsidering ASI. For those working in RL, AGI was never the goal - AlphaGo and AlphaStar already proved that RL can achieve superhuman performance. Now we're just applying this to the real world.
Give AI a measurable goal, let it explore, and improve the model through RL - that's all there is to it.
The future is simply about replicating this process in increasingly complex scenarios.
Like giving AI an aircraft manual and letting it learn flight simulation.
Like having AI write articles that get 100,000+ views
Like having AI develop a TikTok clone app
Like having AI write a Nature paper
Let's look forward to the upcoming developments! Keep pushing forward!