Low Frequency Names Exhibit Bias and Overfitting in Contextualizing Language Models

TL;DR

This paper investigates how low-frequency names in training data cause bias and overfitting in language models like BERT and GPT-2, affecting their representation and bias towards minority groups.

Contribution

It reveals the relationship between name frequency, bias, and overfitting in multiple language models, highlighting the impact on minority and female names.

Findings

Infrequent names are less similar to initial representations.

Lower-frequency minority names are more associated with unpleasantness.

Models rely on less context-informed representations for rare names.

Abstract

We use a dataset of U.S. first names with labels based on predominant gender and racial group to examine the effect of training corpus frequency on tokenization, contextualization, similarity to initial representation, and bias in BERT, GPT-2, T5, and XLNet. We show that predominantly female and non-white names are less frequent in the training corpora of these four language models. We find that infrequent names are more self-similar across contexts, with Spearman's r between frequency and self-similarity as low as -.763. Infrequent names are also less similar to initial representation, with Spearman's r between frequency and linear centered kernel alignment (CKA) similarity to initial representation as high as .702. Moreover, we find Spearman's r between racial bias and name frequency in BERT of .492, indicating that lower-frequency minority group names are more associated with unpleasantness. Representations of infrequent names undergo more processing, but are more self-similar, indicating that models rely on less context-informed representations of uncommon and minority names which are overfit to a lower number of observed contexts.