"You Cannot Sound Like GPT": Signs of language discrimination and resistance in computer science publishing

TL;DR

This study reveals persistent linguistic biases in computer science peer review, showing that ChatGPT's introduction has only minimally shifted reviewer perceptions and highlighting ongoing challenges for non-native English speakers in scientific publishing.

Contribution

The paper uncovers how language bias persists in peer review and examines ChatGPT's limited impact on reducing linguistic discrimination in scientific publishing.

Findings

Significant bias against authors from non-English speaking countries.

Minimal change in language bias after ChatGPT's introduction.

Reviewers use linguistic features and ChatGPT style as indicators of author background.

Abstract

LLMs have been celebrated for their potential to help multilingual scientists publish their research. Rather than interpret LLMs as a solution, we hypothesize their adoption can be an indicator of existing linguistic exclusion in scientific writing. Using the case study of ICLR, an influential, international computer science conference, we examine how peer reviewers critique writing clarity. Analyzing almost 80,000 peer reviews, we find significant bias against authors associated with institutions in countries where English is less widely spoken. We see only a muted shift in the expression of this bias after the introduction of ChatGPT in late 2022. To investigate this unexpectedly minor change, we conduct interviews with 14 conference participants from across five continents. Peer reviewers describe associating certain features of writing with people of certain language backgrounds, and such groups in turn with the quality of scientific work. While ChatGPT masks some signs of language background, reviewers explain that they now use ChatGPT "style" and non-linguistic features as indicators of author demographics. Authors, aware of this development, described the ongoing need to remove features which could expose their "non-native" status to reviewers. Our findings offer insight into the role of ChatGPT in the reproduction of scholarly language ideologies which conflate producers of "good English" with producers of "good science."