GeistBERT: Breathing Life into German NLP

TL;DR

GeistBERT is a German-specific transformer model trained on a large corpus, achieving state-of-the-art results across multiple NLP tasks and outperforming larger models.

Contribution

It introduces a new German language model, GeistBERT, trained on 1.3 TB of data, with optimized architecture and training methods for improved NLP performance.

Findings

Achieved state-of-the-art in GermEval 2018 classification

Outperformed larger models in multiple benchmarks

Demonstrated strong results across diverse NLP tasks

Abstract

Advances in transformer-based language models have highlighted the benefits of language-specific pre-training on high-quality corpora. In this context, German NLP stands to gain from updated architectures and modern datasets tailored to the linguistic characteristics of the German language. GeistBERT seeks to improve German language processing by incrementally training on a diverse corpus and optimizing model performance across various NLP tasks. We pre-trained GeistBERT using fairseq, following the RoBERTa base configuration with Whole Word Masking (WWM), and initialized from GottBERT weights. The model was trained on a 1.3 TB German corpus with dynamic masking and a fixed sequence length of 512 tokens. For evaluation, we fine-tuned the model on standard downstream tasks, including NER (CoNLL 2003, GermEval 2014), text classification (GermEval 2018 coarse/fine, 10kGNAD), and NLI (German XNLI), using $F_1$ score and accuracy as evaluation metrics. GeistBERT achieved strong results across all tasks, leading among base models and setting a new state-of-the-art (SOTA) in GermEval 2018 fine text classification. It also outperformed several larger models, particularly in classification benchmarks. To support research in German NLP, we release GeistBERT under the MIT license.