Bielik-Minitron-7B: Compressing Large Language Models via Structured Pruning and Knowledge Distillation for the Polish Language

TL;DR

This paper presents Bielik-Minitron-7B, a compressed Polish language model achieved through structured pruning and knowledge distillation, maintaining high performance while significantly reducing size and inference costs.

Contribution

The paper introduces a novel two-stage compression method combining structured pruning and knowledge distillation tailored for Polish language models.

Findings

Reduced model size by 33.4% from 11.04B to 7.35B parameters.

Recovered approximately 90% of baseline performance.

Achieved up to 50% inference speedup.

Abstract

This report details the creation of Bielik-Minitron-7B, a compressed 7.35B parameter version of the Bielik-11B-v3.0 model, specifically optimized for European languages. By leveraging a two-stage compression methodology inspired by the NVIDIA Minitron approach, we combined structured hybrid pruning and knowledge distillation to reduce the model's parameter count by 33.4%, from 11.04B to 7.35B. We utilized the NVIDIA Model Optimizer for structural pruning and the NVIDIA NeMo Framework for logit-based distillation for quality recovery. Following distillation, the model underwent a rigorous alignment pipeline consisting of Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO-P), and Reinforcement Learning (GRPO). Our final model successfully recovered approximately 90% of the baseline model's performance while providing up to 50% inference speedup. This approach demonstrates an efficient pathway to create language models for less-represented languages, preserving the original model quality while reducing inference deployment costs.