How User Language Affects Conflict Fatality Estimates in ChatGPT

TL;DR

This study investigates how the language used to query ChatGPT influences conflict fatality estimates, revealing significant biases that could reinforce media narratives and conflicts.

Contribution

It uncovers a novel language bias mechanism in ChatGPT that affects conflict data reporting, highlighting implications for information accuracy and bias propagation.

Findings

ChatGPT provides 27±11% lower fatality estimates in attacker languages.

Evasive answers increase the bias discrepancy.

Language bias may reinforce existing conflicts and media biases.

Abstract

OpenAI's ChatGPT language model has gained popularity as a powerful tool for complex problem-solving and information retrieval. However, concerns arise about the reproduction of biases present in the language-specific training data. In this study, we address this issue in the context of the Israeli-Palestinian and Turkish-Kurdish conflicts. Using GPT-3.5, we employed an automated query procedure to inquire about casualties in specific airstrikes, in both Hebrew and Arabic for the former conflict and Turkish and Kurdish for the latter. Our analysis reveals that GPT-3.5 provides 27$\pm$11 percent lower fatality estimates when queried in the language of the attacker than in the language of the targeted group. Evasive answers denying the existence of such attacks further increase the discrepancy, creating a novel bias mechanism not present in regular search engines. This language bias has the potential to amplify existing media biases and contribute to information bubbles, ultimately reinforcing conflicts.