Controlled Analyses of Social Biases in Wikipedia Bios

TL;DR

This paper introduces a methodology to analyze social biases in Wikipedia biographies by controlling for confounding attributes, revealing biases related to gender and race that might be overlooked otherwise.

Contribution

It presents a novel approach for isolating demographic attributes in Wikipedia bios, enabling more accurate bias detection and analysis.

Findings

Biases in Wikipedia bios vary across demographic groups.

Controlling for covariates changes bias detection results.

Framework aids future bias mitigation efforts.

Abstract

Social biases on Wikipedia, a widely-read global platform, could greatly influence public opinion. While prior research has examined man/woman gender bias in biography articles, possible influences of other demographic attributes limit conclusions. In this work, we present a methodology for analyzing Wikipedia pages about people that isolates dimensions of interest (e.g., gender), from other attributes (e.g., occupation). Given a target corpus for analysis (e.g.~biographies about women), we present a method for constructing a comparison corpus that matches the target corpus in as many attributes as possible, except the target one. We develop evaluation metrics to measure how well the comparison corpus aligns with the target corpus and then examine how articles about gender and racial minorities (cis. women, non-binary people, transgender women, and transgender men; African American, Asian American, and Hispanic/Latinx American people) differ from other articles. In addition to identifying suspect social biases, our results show that failing to control for covariates can result in different conclusions and veil biases. Our contributions include methodology that facilitates further analyses of bias in Wikipedia articles, findings that can aid Wikipedia editors in reducing biases, and a framework and evaluation metrics to guide future work in this area.