Metadata Might Make Language Models Better

TL;DR

Incorporating metadata such as time, politics, and geography into language models trained on historical texts improves their robustness, fairness, and overall performance, as demonstrated through experiments on 19th-century newspapers.

Contribution

This study extends the time-masking approach by systematically evaluating different strategies for integrating metadata into language models trained on historical data.

Findings

Metadata inclusion improves model robustness.

Metadata enhances fairness in language models.

Models with metadata outperform baseline models.

Abstract

This paper discusses the benefits of including metadata when training language models on historical collections. Using 19th-century newspapers as a case study, we extend the time-masking approach proposed by Rosin et al., 2022 and compare different strategies for inserting temporal, political and geographical information into a Masked Language Model. After fine-tuning several DistilBERT on enhanced input data, we provide a systematic evaluation of these models on a set of evaluation tasks: pseudo-perplexity, metadata mask-filling and supervised classification. We find that showing relevant metadata to a language model has a beneficial impact and may even produce more robust and fairer models.