BIPOLAR: Polarization-based granular framework for LLM bias evaluation

TL;DR

This paper introduces BIPOLAR, a flexible framework for evaluating polarization-related biases in large language models using synthetic datasets and sentiment metrics, revealing nuanced biases across models and topics.

Contribution

The study presents a novel, reusable, and topic-agnostic framework combining sentiment analysis with synthetic datasets to assess bias in LLMs, enabling detailed bias pattern analysis.

Findings

Models show a general positive bias toward Ukraine.

Bias varies significantly across semantic categories.

Prompt modifications influence bias levels.

Abstract

Large language models (LLMs) are known to exhibit biases in downstream tasks, especially when dealing with sensitive topics such as political discourse, gender identity, ethnic relations, or national stereotypes. Although significant progress has been made in bias detection and mitigation techniques, certain challenges remain underexplored. This study proposes a reusable, granular, and topic-agnostic framework to evaluate polarisation-related biases in LLM (both open-source and closed-source). Our approach combines polarisation-sensitive sentiment metrics with a synthetically generated balanced dataset of conflict-related statements, using a predefined set of semantic categories. As a case study, we created a synthetic dataset that focusses on the Russia-Ukraine war, and we evaluated the bias in several LLMs: Llama-3, Mistral, GPT-4, Claude 3.5, and Gemini 1.0. Beyond aggregate bias scores, with a general trend for more positive sentiment toward Ukraine, the framework allowed fine-grained analysis with considerable variation between semantic categories, uncovering divergent behavioural patterns among models. Adaptation to prompt modifications showed further bias towards preconceived language and citizenship modification. Overall, the framework supports automated dataset generation and fine-grained bias assessment, is applicable to a variety of polarisation-driven scenarios and topics, and is orthogonal to many other bias-evaluation strategies.