Abstract:In this paper, we present TransMLA, a framework that seamlessly converts any GQA-based pre-trained model into an MLA-based model. Our approach enables direct compatibility with DeepSeek's codebase, allowing these models to fully leverage DeepSeek-specific optimizations such as vLLM and SGlang. By compressing 93% of the KV cache in LLaMA-2-7B, TransMLA achieves a 10.6x inference speedup at an 8K context length while preserving meaningful output quality. Additionally, the model requires only 6 billion tokens for fine-tuning to regain performance on par with the original across multiple benchmarks. TransMLA offers a practical solution for migrating GQA-based models to the MLA structure. When combined with DeepSeek's advanced features, such as FP8 quantization and Multi-Token Prediction, even greater inference acceleration can be realized.
Submission history
From: Meng Fanxu [view email]
[v1]
Tue, 11 Feb 2025 18:20:18 UTC (326 KB)
[v2]
Thu, 13 Feb 2025 18:07:04 UTC (327 KB)
[v3]
Tue, 27 May 2025 05:11:57 UTC (3,706 KB)
[v4]
Wed, 28 May 2025 12:07:57 UTC (3,706 KB)
[v5]
Thu, 12 Jun 2025 11:45:57 UTC (429 KB)