Do they mean 'us'? Interpreting Referring Expressions in Intergroup Bias

TL;DR

This paper investigates intergroup bias in sports commentary by modeling it as a tagging task, using a large dataset and large language models to analyze how language reflects group identity and bias.

Contribution

It introduces a novel dataset of NFL game comments with annotations for referring expressions and demonstrates large language models' effectiveness in analyzing intergroup bias.

Findings

LLMs perform best when prompted with linguistic descriptions of win probabilities.

Linear variations in referent usage distinguish in-group and out-group comments.

Expert annotations validate the modeling approach.

Abstract

The variations between in-group and out-group speech (intergroup bias) are subtle and could underlie many social phenomena like stereotype perpetuation and implicit bias. In this paper, we model the intergroup bias as a tagging task on English sports comments from forums dedicated to fandom for NFL teams. We curate a unique dataset of over 6 million game-time comments from opposing perspectives (the teams in the game), each comment grounded in a non-linguistic description of the events that precipitated these comments (live win probabilities for each team). Expert and crowd annotations justify modeling the bias through tagging of implicit and explicit referring expressions and reveal the rich, contextual understanding of language and the world required for this task. For large-scale analysis of intergroup variation, we use LLMs for automated tagging, and discover that some LLMs perform best when prompted with linguistic descriptions of the win probability at the time of the comment, rather than numerical probability. Further, large-scale tagging of comments using LLMs uncovers linear variations in the form of referent across win probabilities that distinguish in-group and out-group utterances. Code and data are available at https://github.com/venkatasg/intergroup-nfl .