Mistral 7B

TL;DR

Mistral 7B is a new 7-billion-parameter language model that outperforms larger models in various benchmarks through innovative attention mechanisms and instruction tuning, offering efficient inference and high performance.

Contribution

Introduces Mistral 7B with grouped-query and sliding window attention, achieving superior performance and efficiency over larger models, and provides an instruction-tuned variant surpassing comparable models.

Findings

Outperforms Llama 2 13B across all benchmarks

Surpasses Llama 1 34B in reasoning, math, and code generation

Efficient inference with GQA and SWA mechanisms

Abstract

We introduce Mistral 7B v0.1, a 7-billion-parameter language model engineered for superior performance and efficiency. Mistral 7B outperforms Llama 2 13B across all evaluated benchmarks, and Llama 1 34B in reasoning, mathematics, and code generation. Our model leverages grouped-query attention (GQA) for faster inference, coupled with sliding window attention (SWA) to effectively handle sequences of arbitrary length with a reduced inference cost. We also provide a model fine-tuned to follow instructions, Mistral 7B -- Instruct, that surpasses the Llama 2 13B -- Chat model both on human and automated benchmarks. Our models are released under the Apache 2.0 license.