A Systematic Analysis of Biases in Large Language Models

TL;DR

This paper systematically examines biases in four large language models across politics, ideology, language, and gender, revealing that despite efforts for neutrality, biases and inclinations persist in these models.

Contribution

It provides a comprehensive analysis of biases in multiple LLMs across diverse social and political dimensions, highlighting areas needing improvement.

Findings

Models show biases in political neutrality and ideological stance.

Language biases are evident in multilingual story completion.

Gender-related affinities are present in model responses.

Abstract

Large language models (LLMs) have rapidly become indispensable tools for acquiring information and supporting human decision-making. However, ensuring that these models uphold fairness across varied contexts is critical to their safe and responsible deployment. In this study, we undertake a comprehensive examination of four widely adopted LLMs, probing their underlying biases and inclinations across the dimensions of politics, ideology, alliance, language, and gender. Through a series of carefully designed experiments, we investigate their political neutrality using news summarization, ideological biases through news stance classification, tendencies toward specific geopolitical alliances via United Nations voting patterns, language bias in the context of multilingual story completion, and gender-related affinities as revealed by responses to the World Values Survey. Results indicate that while the LLMs are aligned to be neutral and impartial, they still show biases and affinities of different types.