LLaMA: Open and Efficient Foundation Language Models

TL;DR

LLaMA introduces a series of open, efficient foundation language models trained on publicly available data, achieving competitive performance with larger proprietary models and promoting accessible AI research.

Contribution

The paper presents LLaMA, a new set of open foundation models trained solely on public data, outperforming or matching larger proprietary models on key benchmarks.

Findings

LLaMA-13B outperforms GPT-3 (175B) on most benchmarks.

LLaMA-65B is competitive with Chinchilla-70B and PaLM-540B.

All models are publicly released for research use.

Abstract

We introduce LLaMA, a collection of foundation language models ranging from 7B to 65B parameters. We train our models on trillions of tokens, and show that it is possible to train state-of-the-art models using publicly available datasets exclusively, without resorting to proprietary and inaccessible datasets. In particular, LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, and LLaMA-65B is competitive with the best models, Chinchilla-70B and PaLM-540B. We release all our models to the research community.