Evaluating Short-Term Temporal Fluctuations of Social Biases in Social Media Data and Masked Language Models

TL;DR

This paper investigates how social biases in Masked Language Models evolve over time by analyzing models trained on chronological social media data, finding that most biases remain stable despite increasing data volume.

Contribution

It provides the first empirical analysis of temporal fluctuations of social biases in MLMs trained on social media data over time.

Findings

Most social biases in MLMs are stable over time.

Certain demographic preferences, like male over female, persist consistently.

Biases do not significantly amplify with more training data.

Abstract

Social biases such as gender or racial biases have been reported in language models (LMs), including Masked Language Models (MLMs). Given that MLMs are continuously trained with increasing amounts of additional data collected over time, an important yet unanswered question is how the social biases encoded with MLMs vary over time. In particular, the number of social media users continues to grow at an exponential rate, and it is a valid concern for the MLMs trained specifically on social media data whether their social biases (if any) would also amplify over time. To empirically analyse this problem, we use a series of MLMs pretrained on chronologically ordered temporal snapshots of corpora. Our analysis reveals that, although social biases are present in all MLMs, most types of social bias remain relatively stable over time (with a few exceptions). To further understand the mechanisms that influence social biases in MLMs, we analyse the temporal corpora used to train the MLMs. Our findings show that some demographic groups, such as male, obtain higher preference over the other, such as female on the training corpora constantly.