How BERT Speaks Shakespearean English? Evaluating Historical Bias in Contextual Language Models

TL;DR

This study evaluates how well BERT and other models understand Early Modern and Modern English, revealing biases and differences in their language comprehension through fill-in-the-blank tests.

Contribution

It introduces a method to measure historical language bias in contextual models using targeted fill-in-the-blank tasks and scoring.

Findings

Models show varying adequacy for EME and ME English

BERT exhibits certain biases toward modern language forms

Evaluation framework can quantify historical language understanding

Abstract

In this paper, we explore the idea of analysing the historical bias of contextual language models based on BERT by measuring their adequacy with respect to Early Modern (EME) and Modern (ME) English. In our preliminary experiments, we perform fill-in-the-blank tests with 60 masked sentences (20 EME-specific, 20 ME-specific and 20 generic) and three different models (i.e., BERT Base, MacBERTh, English HLM). We then rate the model predictions according to a 5-point bipolar scale between the two language varieties and derive a weighted score to measure the adequacy of each model to EME and ME varieties of English.