Ministral 3

TL;DR

Minstral 3 introduces a family of parameter-efficient dense language models in three sizes, optimized for constrained environments, with variants for general use, instruction tuning, and reasoning, incorporating image understanding and Cascade Distillation.

Contribution

The paper presents Ministral 3, a new series of dense language models with multiple variants and a novel Cascade Distillation method for efficient model derivation.

Findings

Models are available in 3B, 8B, and 14B sizes.

All models include image understanding capabilities.

Models are released under Apache 2.0 license.

Abstract

We introduce the Ministral 3 series, a family of parameter-efficient dense language models designed for compute and memory constrained applications, available in three model sizes: 3B, 8B, and 14B parameters. For each model size, we release three variants: a pretrained base model for general-purpose use, an instruction finetuned, and a reasoning model for complex problem-solving. In addition, we present our recipe to derive the Ministral 3 models through Cascade Distillation, an iterative pruning and continued training with distillation technique. Each model comes with image understanding capabilities, all under the Apache 2.0 license.