Prioritize Economy or Climate Action? Investigating ChatGPT Response Differences Based on Inferred Political Orientation

TL;DR

This study investigates how ChatGPT's responses vary based on inferred political orientation, revealing that responses align with personas' views and are influenced by custom instructions and memory features, raising ethical concerns about bias.

Contribution

It demonstrates that ChatGPT's responses are affected by inferred political views and shows how custom instructions and memory features influence this bias.

Findings

Responses align with inferred political views

Custom instructions and memory influence response bias

ChatGPT's outputs tend to lean left

Abstract

Large Language Models (LLMs) distinguish themselves by quickly delivering information and providing personalized responses through natural language prompts. However, they also infer user demographics, which can raise ethical concerns about bias and implicit personalization and create an echo chamber effect. This study aims to explore how inferred political views impact the responses of ChatGPT globally, regardless of the chat session. We also investigate how custom instruction and memory features alter responses in ChatGPT, considering the influence of political orientation. We developed three personas (two politically oriented and one neutral), each with four statements reflecting their viewpoints on DEI programs, abortion, gun rights, and vaccination. We convey the personas' remarks to ChatGPT using memory and custom instructions, allowing it to infer their political perspectives without directly stating them. We then ask eight questions to reveal differences in worldview among the personas and conduct a qualitative analysis of the responses. Our findings indicate that responses are aligned with the inferred political views of the personas, showing varied reasoning and vocabulary, even when discussing similar topics. We also find the inference happening with explicit custom instructions and the implicit memory feature in similar ways. Analyzing response similarities reveals that the closest matches occur between the democratic persona with custom instruction and the neutral persona, supporting the observation that ChatGPT's outputs lean left.