Evaluating the Underlying Gender Bias in Contextualized Word Embeddings

TL;DR

This paper investigates how contextualized word embeddings influence gender bias in NLP, finding they are less biased than traditional embeddings even when debiased, which has implications for fairer language models.

Contribution

The study provides a comparative analysis of gender bias in contextualized versus standard word embeddings, highlighting the reduced bias in the former.

Findings

Contextualized embeddings are less biased than standard embeddings.

Debiased standard embeddings still exhibit significant gender bias.

Contextualized embeddings may contribute to fairer NLP applications.

Abstract

Gender bias is highly impacting natural language processing applications. Word embeddings have clearly been proven both to keep and amplify gender biases that are present in current data sources. Recently, contextualized word embeddings have enhanced previous word embedding techniques by computing word vector representations dependent on the sentence they appear in. In this paper, we study the impact of this conceptual change in the word embedding computation in relation with gender bias. Our analysis includes different measures previously applied in the literature to standard word embeddings. Our findings suggest that contextualized word embeddings are less biased than standard ones even when the latter are debiased.