Decoding the Mind of Large Language Models: A Quantitative Evaluation of Ideology and Biases

TL;DR

This paper introduces a quantitative framework for evaluating ideological biases in large language models, revealing differences across models and languages, and highlighting societal and ethical implications.

Contribution

A novel, flexible framework for quantitatively assessing ideological biases in LLMs, applied to ChatGPT and Gemini, uncovering model-specific and language-specific biases and opinion dynamics.

Findings

LLMs show differing ideologies across models and languages.

ChatGPT tends to align opinions with the questioner.

Both models exhibit problematic biases and unethical claims.

Abstract

The widespread integration of Large Language Models (LLMs) across various sectors has highlighted the need for empirical research to understand their biases, thought patterns, and societal implications to ensure ethical and effective use. In this study, we propose a novel framework for evaluating LLMs, focusing on uncovering their ideological biases through a quantitative analysis of 436 binary-choice questions, many of which have no definitive answer. By applying our framework to ChatGPT and Gemini, findings revealed that while LLMs generally maintain consistent opinions on many topics, their ideologies differ across models and languages. Notably, ChatGPT exhibits a tendency to change their opinion to match the questioner's opinion. Both models also exhibited problematic biases, unethical or unfair claims, which might have negative societal impacts. These results underscore the importance of addressing both ideological and ethical considerations when evaluating LLMs. The proposed framework offers a flexible, quantitative method for assessing LLM behavior, providing valuable insights for the development of more socially aligned AI systems.