County-level Algorithmic Audit of Racial Bias in Twitter's Home Timeline

TL;DR

This paper conducts a large-scale audit of Twitter's Home Timeline to investigate racial bias by analyzing county-level data, revealing systemic disparities in Tweet visibility among racial groups without direct user racial information.

Contribution

It introduces a novel county-based methodology to assess racial bias on Twitter, overcoming the lack of individual racial data.

Findings

County racial composition correlates with Tweet visibility disparities.

First large-scale audit of racial bias on Twitter.

Highlights challenges in measuring bias without user racial data.

Abstract

We report on the outcome of an audit of Twitter's Home Timeline ranking system. The goal of the audit was to determine if authors from some racial groups experience systematically higher impression counts for their Tweets than others. A central obstacle for any such audit is that Twitter does not ordinarily collect or associate racial information with its users, thus prohibiting an analysis at the level of individual authors. Working around this obstacle, we take US counties as our unit of analysis. We associate each user in the United States on the Twitter platform to a county based on available location data. The US Census Bureau provides information about the racial decomposition of the population in each county. The question we investigate then is if the racial decomposition of a county is associated with the visibility of Tweets originating from within the county. Focusing on two racial groups, the Black or African American population and the White population as defined by the US Census Bureau, we evaluate two statistical measures of bias. Our investigation represents the first large-scale algorithmic audit into racial bias on the Twitter platform. Additionally, it illustrates the challenges of measuring racial bias in online platforms without having such information on the users.