Unfair Mistakes on Social Media: How Demographic Characteristics influence Authorship Attribution

TL;DR

This study systematically audits authorship attribution models on social media for demographic bias, revealing that while models seem fair in closed settings, errors tend to favor users sharing demographic traits, highlighting fairness issues in real-world scenarios.

Contribution

The paper provides a comprehensive bias audit of authorship attribution models across multiple demographic groups, revealing nuanced fairness issues especially when true authors are excluded from candidate sets.

Findings

Authorship attribution models show no bias in closed-world settings.

Errors tend to favor users sharing demographic traits with the true author.

Fairness in closed settings does not guarantee fairness in open-world error scenarios.

Abstract

Authorship attribution techniques are increasingly being used in online contexts such as sock puppet detection, malicious account linking, and cross-platform account linking. Yet, it is unknown whether these models perform equitably across different demographic groups. Bias in such techniques could lead to false accusations, account banning, and privacy violations disproportionately impacting users from certain demographics. In this paper, we systematically audit authorship attribution for bias with respect to gender, native language, and age. We evaluate fairness in 3 ways. First, we evaluate how the proportion of users with a certain demographic characteristic impacts the overall classifier performance. Second, we evaluate if a user's demographic characteristics influence the probability that their texts are misclassified. Our analysis indicates that authorship attribution does not demonstrate bias across demographic groups in the closed-world setting. Third, we evaluate the types of errors that occur when the true author is removed from the suspect set, thereby forcing the classifier to choose an incorrect author. Unlike the first two settings, this analysis demonstrates a tendency to attribute authorship to users who share the same demographic characteristic as the true author. Crucially, these errors do not only include texts that deviate from a user's usual style, but also those that are very close to the author's average. Our results highlight that though a model may appear fair in the closed-world setting for a performant classifier, this does not guarantee fairness when errors are inevitable.