Mistral Large 4

· arXiv ·

5 min read Original article ↗

A coding agent working through a large repository needs to keep distant parts of the task in view, but storing and processing all of that context is expensive to serve. Mistral Large 4's developers first adopted aggressive memory-saving tricks, found that the resulting models looked fine on standard checks but fell short on long agentic work, and then restored long-context access in more layers. They controlled the resulting cost by teaching the model to select which past tokens to attend to. The telling observation is that models with comparable language-modeling loss could score differently on SWE-bench Verified, a software-engineering benchmark.

SWE-bench Verified score versus validation perplexity for attention architecture variants
Right panel of Figure 5: SWE-bench Verified after midtraining plotted against validation perplexity for several attention variants. Architectures at comparable perplexity land at different agentic scores. Per the figure notes, the first two points are excluded.

The paper describes this as architecture choices imposing an agentic penalty even at comparable language-modeling loss. Mistral Large 4 itself is a multimodal mixture-of-experts model with 52B active parameters and 1T total parameters.

The initial design saved serving memory in several stacked ways. Ragged multi-head latent attention used full attention in one of every four layers and a 512-token sliding window in the others. Cross-layer attention (CLA) let a layer reuse the key/value (KV) cache of a preceding full-attention layer. YOCO let global-attention layers in the second half of the network reuse KV values from the first half. Together these cut KV-cache size by a factor of 16 versus a full-attention baseline, and YOCO also enabled a two-stage prefilling that the authors measured as 1.7× faster at 1M context on their benchmarks. Early checks showed little or acceptable impact on common pretraining and post-training evaluations, but later supervised fine-tuning evaluations were weaker than expected on long-context agentic tasks.

Figure 5 pairs two views of this. The left panel plots MMLU Redux accuracy against monorepo perplexity during pretraining, and the relationship is comparatively close across the architecture variants. The right panel, shown above, plots SWE-bench Verified against validation perplexity, where results spread out at similar perplexity. The authors did not observe the same gap on knowledge-based evaluations, and they say longer training can mitigate the agentic penalty. This is a comparison across attention variants, so it shows that perplexity and knowledge scores missed something, not that attention is the only cause.

To recover long-range capability, the team performed "model surgery" on a trained model. It removed the sliding-window limits so affected layers could attend to the full context, and it removed CLA sharing by adding KV-producing weights copied from a preceding long-attention layer. At smaller compute scales it also tested broader removal of sharing, including YOCO. For the full-scale model it kept YOCO: at the larger test scale, surgery followed by midtraining alone favored retaining it, though the authors could not rerun the full phase-2-plus-midtraining schedule for that test. The final architecture drops sliding windows and CLA, keeps YOCO sharing among the original global-attention layers in the second half, and adds LeRoPE and SSMax. Keeping YOCO is a qualified choice for this model, not a general verdict on the technique.

Restoring full attention brings back the cost, so the authors reintroduced sparsity with DeepSeek Sparse Attention (DSA). A lightweight indexer picks the top 2,048 key tokens for each query, so attention works on a selected subset of the past. The indexers were first warmed up for about 15.1B tokens with the main model frozen and using full attention, distilling its attention distributions. Then indexers and main model were trained jointly under sparse attention for another 198B tokens. Former sliding-window layers share an indexer within specified groups to reduce overhead. The sparse stage is presented as a way to cut long-context attention cost, not as a demonstrated agentic gain over full attention. The closest prior work is DeepSeek-V3.2-Exp, which introduced DSA via continued training and reported long-context efficiency gains with broadly similar performance to DeepSeek-V3.1-Terminus; that comparison is between those DeepSeek models, not against Mistral Large 4. (DeepSeek-V3.2-Exp report)

Agentic training also depends on data. Agents turn seeds into candidate executable tasks, which pass cheap checks (LLM judges, polarity checks), then costlier ones (an agentic verifier, reward-hacking checks), then difficulty calibration. Failures can return to the seed with feedback, though seeds that fail the initial filter are rejected. The authors note that agents remain fallible and some domains need post-hoc adjustment and filtering.

Synthetic task generation and verification workflow
The authors' conceptual pipeline for synthetic tasks: a seed is filtered, a candidate is generated, and it passes cheap then expensive checks and difficulty calibration before acceptance. Failures loop back to the seed with feedback.

On benchmarks, the paper's table reports 19.7% on FrontierCode 2.0 for Mistral Large 4 versus 19.3% for DeepSeek V4.1-Flash 1017. That is a score on one named benchmark, and the architecture experiments do not isolate how much of it the attention changes explain.

For deployment, the authors quantize the model to a calibrated, mixed-precision NVFP4 checkpoint of 606 GB, which they say matches the FP8 reference on evaluations and allows serving on four GB300 GPUs.

Decode interactivity versus output throughput for serving optimizations
Decode interactivity (tokens/s/request) against output throughput (tokens/s/GPU) at 50K input length, with points labeled by concurrent-request count. The baseline is an FP8 model with off-the-shelf context parallelism on 8×GB300; NVFP4 is served on 4×GB300.

Under those conditions, optimized decode context parallelism raised decode interactivity by 51% at eight concurrent requests, and DSpark raised it by 90% at eight concurrent requests versus a baseline without speculative decoding. These are serving-speed results, not reductions in generated tokens or agent attempts.

The main lesson is that language-modeling loss alone did not predict the long-horizon coding result in these experiments. The authors therefore used agentic evaluations to steer architecture decisions and treated sparse attention as a later efficiency step, not a substitute for measuring agent performance. The limits are real: the evidence comes from controlled-scale ablations and surgery experiments, the full-scale YOCO test had a limited recovery schedule, and the final model's score gain is not attributed to DSA.