Analyzing Correlations Between Intrinsic and Extrinsic Bias Metrics of Static Word Embeddings With Their Measuring Biases Aligned

TL;DR

This paper investigates how well intrinsic bias metrics of static word embeddings predict biased behaviors in NLP systems by analyzing correlations with extrinsic bias metrics, revealing variable predictive power across different settings.

Contribution

It introduces a method to align intrinsic and extrinsic bias metrics by extracting characteristic words, clarifying when intrinsic metrics effectively predict bias.

Findings

Moderate to high correlation with some extrinsic bias metrics.

Little to no correlation with other extrinsic bias metrics.

Intrinsic bias metrics can predict bias in specific contexts.

Abstract

We examine the abilities of intrinsic bias metrics of static word embeddings to predict whether Natural Language Processing (NLP) systems exhibit biased behavior. A word embedding is one of the fundamental NLP technologies that represents the meanings of words through real vectors, and problematically, it also learns social biases such as stereotypes. An intrinsic bias metric measures bias by examining a characteristic of vectors, while an extrinsic bias metric checks whether an NLP system trained with a word embedding is biased. A previous study found that a common intrinsic bias metric usually does not correlate with extrinsic bias metrics. However, the intrinsic and extrinsic bias metrics did not measure the same bias in most cases, which makes us question whether the lack of correlation is genuine. In this paper, we extract characteristic words from datasets of extrinsic bias metrics and analyze correlations with intrinsic bias metrics with those words to ensure both metrics measure the same bias. We observed moderate to high correlations with some extrinsic bias metrics but little to no correlations with the others. This result suggests that intrinsic bias metrics can predict biased behavior in particular settings but not in others. Experiment codes are available at GitHub.