On the Interpretability and Significance of Bias Metrics in Texts: a PMI-based Approach

TL;DR

This paper introduces a PMI-based bias metric for texts that offers greater interpretability and statistical significance estimation, aligning with embedding-based methods in measuring real-world biases.

Contribution

It proposes a transparent PMI-based bias metric expressed through conditional probabilities, with an approximation to odds ratios for significance testing.

Findings

The PMI-based metric aligns with embedding-based bias measurements.

It allows estimation of confidence intervals for bias significance.

The approach effectively captures real-world gender biases in large corpora.

Abstract

In recent years, word embeddings have been widely used to measure biases in texts. Even if they have proven to be effective in detecting a wide variety of biases, metrics based on word embeddings lack transparency and interpretability. We analyze an alternative PMI-based metric to quantify biases in texts. It can be expressed as a function of conditional probabilities, which provides a simple interpretation in terms of word co-occurrences. We also prove that it can be approximated by an odds ratio, which allows estimating confidence intervals and statistical significance of textual biases. This approach produces similar results to metrics based on word embeddings when capturing gender gaps of the real world embedded in large corpora.