Bielik 11B v3: Multilingual Large Language Model for European Languages

TL;DR

Bielik 11B v3 is a multilingual European language model optimized for Polish, achieving state-of-the-art performance with fewer parameters through a comprehensive training pipeline and efficient deployment options.

Contribution

The paper introduces Bielik 11B v3, a resource-efficient, high-performance multilingual model for European languages, extending the Mistral 7B architecture with novel training strategies.

Findings

Outperforms larger models on diverse tasks

Significantly surpasses specialized Polish models

Offers effective deployment across hardware configurations

Abstract

We present Bielik 11B v3, a state-of-the-art language model highly optimized for the Polish language, while also maintaining strong capabilities in other European languages. This model extends the Mistral 7B v0.2 architecture, scaled to 11B parameters via depth up-scaling. Its development involved a comprehensive four-stage training pipeline: continuous pre-training, supervised fine-tuning (SFT), Direct Preference Optimization (DPO), and reinforcement learning. Comprehensive evaluations demonstrate that Bielik 11B v3 achieves exceptional performance. It significantly surpasses other specialized Polish language models and outperforms many larger models (with 2-6 times more parameters) on a wide range of tasks, from basic linguistic understanding to complex reasoning. The model's parameter efficiency, combined with extensive quantization options, allows for effective deployment across diverse hardware configurations. Bielik 11B v3 not only advances AI capabilities for the Polish language but also establishes a new benchmark for developing resource-efficient, high-performance models for less-represented languages.