Finnish Language Modeling with Deep Transformer Models

TL;DR

This paper evaluates Transformer-based models, BERT and Transformer-XL, for Finnish language modeling, demonstrating significant improvements over previous LSTM models in perplexity scores.

Contribution

It provides the first pseudo-perplexity score for Finnish BERT and shows Transformer-XL outperforms LSTM models by 27% in perplexity.

Findings

BERT achieves a pseudo-perplexity of 14.5.

Transformer-XL improves perplexity to 73.58.

Transformer models outperform LSTM in Finnish language modeling.

Abstract

Transformers have recently taken the center stage in language modeling after LSTM's were considered the dominant model architecture for a long time. In this project, we investigate the performance of the Transformer architectures-BERT and Transformer-XL for the language modeling task. We use a sub-word model setting with the Finnish language and compare it to the previous State of the art (SOTA) LSTM model. BERT achieves a pseudo-perplexity score of 14.5, which is the first such measure achieved as far as we know. Transformer-XL improves upon the perplexity score to 73.58 which is 27\% better than the LSTM model.