Argumentative Experience: Reducing Confirmation Bias on Controversial Issues through LLM-Generated Multi-Persona Debates

TL;DR

This study evaluates LLM-powered multi-persona debates' effectiveness in reducing confirmation bias, finding limited impact on belief change but some buffering against cognitive bias, with insights for future design improvements.

Contribution

It provides empirical evidence on the effectiveness of multi-persona debate systems in mitigating confirmation bias and offers design insights for enhancing balanced information engagement.

Findings

No significant increase in attention to opposing views

Limited belief change observed

Buffering effect against cognitive bias

Abstract

Multi-persona debate systems powered by large language models (LLMs) show promise in reducing confirmation bias, which can fuel echo chambers and social polarization. However, empirical evidence remains limited on whether they meaningfully shift user attention toward belief-challenging content, promote belief change, or outperform traditional debiasing strategies. To investigate this, we compare an LLM-based multi-persona debate system with a two-stance retrieval-based system, exposing participants to multiple viewpoints on controversial topics. By collecting eye-tracking data, belief change measures, and qualitative feedback, our results show that while the debate system does not significantly increase attention to opposing views, or make participants shift away from prior beliefs, it does provide a buffering effect against bias caused by individual cognitive tendency. These findings shed light on both the promise and limits of multi-persona debate systems in information seeking, and we offer design insights to guide future work toward more balanced and reflective information engagement.