PoliCon: Evaluating LLMs on Achieving Diverse Political Consensus Objectives

TL;DR

PoliCon is a new benchmark using European Parliament records to evaluate large language models' ability to draft political consensus resolutions, revealing their limitations and biases in complex decision-making scenarios.

Contribution

Introduces PoliCon, a comprehensive benchmark and evaluation framework for assessing LLMs' capacity to generate political consensus resolutions based on diverse parliamentary contexts.

Findings

State-of-the-art LLMs struggle with complex consensus tasks.

Models exhibit partisan biases and favor dominant parties.

PoliCon effectively reveals LLMs' strengths and weaknesses in political decision-making.

Abstract

Achieving political consensus is crucial yet challenging for the effective functioning of social governance. However, although frontier AI systems represented by large language models (LLMs) have developed rapidly in recent years, their capabilities in this scope are still understudied. In this paper, we introduce PoliCon, a novel benchmark constructed from 2,225 high-quality deliberation records of the European Parliament over 13 years, ranging from 2009 to 2022, to evaluate the ability of LLMs to draft consensus resolutions based on divergent party positions under varying collective decision-making contexts and political requirements. Specifically, PoliCon incorporates four factors to build each task environment for finding different political consensus: specific political issues, political goals, participating parties, and power structures based on seat distribution. We also developed an evaluation framework based on social choice theory for PoliCon, which simulates the real voting outcomes of different political parties to assess whether LLM-generated resolutions meet the requirements of the predetermined political consensus. Our experimental results demonstrate that even state-of-the-art models remain undersatisfied with complex tasks like passing resolutions by a two-thirds majority and addressing security issues, while uncovering their inherent partisan biases and revealing some behaviors LLMs show to achieve the consensus, such as prioritizing the stance of the dominant party instead of uniting smaller parties, which highlights PoliCon's promise as an effective platform for studying LLMs' ability to promote political consensus. The code and dataset are released at https://zowiezhang.github.io/projects/PoliCon.