Abstract:We present TinyLlama, a compact 1.1B language model pretrained on around 1 trillion tokens for approximately 3 epochs. Building on the architecture and tokenizer of Llama 2, TinyLlama leverages various advances contributed by the open-source community (e.g., FlashAttention and Lit-GPT), achieving better computational efficiency. Despite its relatively small size, TinyLlama demonstrates remarkable performance in a series of downstream tasks. It significantly outperforms existing open-source language models with comparable sizes. Our model checkpoints and code are publicly available on GitHub at this https URL.
Submission history
From: Guangtao Zeng [view email]
[v1]
Thu, 4 Jan 2024 17:54:59 UTC (1,783 KB)
[v2]
Tue, 4 Jun 2024 02:05:30 UTC (2,366 KB)