PLLuM: A Family of Polish Large Language Models

TL;DR

PLLuM introduces the largest open-source Polish language models, developed with a new extensive corpus and safety frameworks, to enhance AI support for Polish and promote open research.

Contribution

It presents a new family of Polish language models with a large corpus, safety measures, and alignment techniques, filling a gap in non-English AI resources.

Findings

Models perform well on downstream tasks in public administration.

Introduction of a comprehensive safety and governance framework.

Open release to support research and sovereignty in Poland.

Abstract

Large Language Models (LLMs) play a central role in modern artificial intelligence, yet their development has been primarily focused on English, resulting in limited support for other languages. We present PLLuM (Polish Large Language Model), the largest open-source family of foundation models tailored specifically for the Polish language. Developed by a consortium of major Polish research institutions, PLLuM addresses the need for high-quality, transparent, and culturally relevant language models beyond the English-centric commercial landscape. We describe the development process, including the construction of a new 140-billion-token Polish text corpus for pre-training, a 77k custom instructions dataset, and a 100k preference optimization dataset. A key component is a Responsible AI framework that incorporates strict data governance and a hybrid module for output correction and safety filtering. We detail the models' architecture, training procedures, and alignment techniques for both base and instruction-tuned variants, and demonstrate their utility in a downstream task within public administration. By releasing these models publicly, PLLuM aims to foster open research and strengthen sovereign AI technologies in Poland.