Red AI? Inconsistent Responses from GPT3.5 Models on Political Issues in the US and China

TL;DR

This study reveals that GPT's multilingual models exhibit significant political bias and inconsistency in responses on US and China issues, influenced by language and geopolitical factors, raising concerns about information reliability.

Contribution

It is the first to analyze political biases in GPT's multilingual responses, highlighting language-based disparities and potential influence of censorship and geopolitics.

Findings

Chinese GPT responses are more pro-China and less negative towards China's issues.

English GPT responses are more negative towards China.

Responses show bias towards the issue's 'own' country based on language used.

Abstract

The rising popularity of ChatGPT and other AI-powered large language models (LLMs) has led to increasing studies highlighting their susceptibility to mistakes and biases. However, most of these studies focus on models trained on English texts. Taking an innovative approach, this study investigates political biases in GPT's multilingual models. We posed the same question about high-profile political issues in the United States and China to GPT in both English and simplified Chinese, and our analysis of the bilingual responses revealed that GPT's bilingual models' political "knowledge" (content) and the political "attitude" (sentiment) are significantly more inconsistent on political issues in China. The simplified Chinese GPT models not only tended to provide pro-China information but also presented the least negative sentiment towards China's problems, whereas the English GPT was significantly more negative towards China. This disparity may stem from Chinese state censorship and US-China geopolitical tensions, which influence the training corpora of GPT bilingual models. Moreover, both Chinese and English models tended to be less critical towards the issues of "their own" represented by the language used, than the issues of "the other." This suggests that GPT multilingual models could potentially develop a "political identity" and an associated sentiment bias based on their training language. We discussed the implications of our findings for information transmission and communication in an increasingly divided world.