TinyLlama: An Open-Source Small Language Model

TL;DR

TinyLlama is a 1.1-billion-parameter open-source language model that, through efficient training and architecture, achieves strong performance on downstream tasks, surpassing similar-sized open-source models.

Contribution

It introduces TinyLlama, a small, efficient language model built on Llama 2 architecture, with improved training techniques and community-driven advances, outperforming comparable open-source models.

Findings

Outperforms existing open-source models of similar size

Achieves strong results on various downstream tasks

Demonstrates high computational efficiency

Abstract

We present TinyLlama, a compact 1.1B language model pretrained on around 1 trillion tokens for approximately 3 epochs. Building on the architecture and tokenizer of Llama 2, TinyLlama leverages various advances contributed by the open-source community (e.g., FlashAttention and Lit-GPT), achieving better computational efficiency. Despite its relatively small size, TinyLlama demonstrates remarkable performance in a series of downstream tasks. It significantly outperforms existing open-source language models with comparable sizes. Our model checkpoints and code are publicly available on GitHub at https://github.com/jzhang38/TinyLlama.