Pre-training Polish Transformer-based Language Models at Scale

TL;DR

This paper introduces large-scale Polish transformer-based language models trained on extensive datasets, demonstrating significant improvements across multiple NLP tasks compared to previous models.

Contribution

The authors present two new Polish BERT-based models trained on over 1 billion sentences, with detailed methodology and evaluation showing superior performance.

Findings

Models outperform previous approaches on 11 out of 13 tasks

Training on large-scale data improves NLP performance for Polish

Methodology for data collection and pre-training is detailed

Abstract

Transformer-based language models are now widely used in Natural Language Processing (NLP). This statement is especially true for English language, in which many pre-trained models utilizing transformer-based architecture have been published in recent years. This has driven forward the state of the art for a variety of standard NLP tasks such as classification, regression, and sequence labeling, as well as text-to-text tasks, such as machine translation, question answering, or summarization. The situation have been different for low-resource languages, such as Polish, however. Although some transformer-based language models for Polish are available, none of them have come close to the scale, in terms of corpus size and the number of parameters, of the largest English-language models. In this study, we present two language models for Polish based on the popular BERT architecture. The larger model was trained on a dataset consisting of over 1 billion polish sentences, or 135GB of raw text. We describe our methodology for collecting the data, preparing the corpus, and pre-training the model. We then evaluate our models on thirteen Polish linguistic tasks, and demonstrate improvements over previous approaches in eleven of them.