Arbiters of Ambivalence: Challenges of Using LLMs in No-Consensus Tasks

TL;DR

This paper investigates the limitations of large language models in handling ambivalent scenarios, revealing they often take sides in no-consensus topics when acting as judges or debaters, highlighting challenges in alignment.

Contribution

It introduces a no-consensus benchmark and analyzes LLMs' biases across different roles, revealing their tendency to choose sides in ambiguous situations.

Findings

LLMs can generate nuanced answers but often take a stance in no-consensus topics.

LLMs struggle to reflect human disagreement in judgment roles.

Highlighting the need for advanced alignment methods without human oversight.

Abstract

The increasing use of LLMs as substitutes for humans in ``aligning'' LLMs has raised questions about their ability to replicate human judgments and preferences, especially in ambivalent scenarios where humans disagree. This study examines the biases and limitations of LLMs in three roles: answer generator, judge, and debater. These roles loosely correspond to previously described alignment frameworks: preference alignment (judge) and scalable oversight (debater), with the answer generator reflecting the typical setting with user interactions. We develop a ``no-consensus'' benchmark by curating examples that encompass a variety of a priori ambivalent scenarios, each presenting two possible stances. Our results show that while LLMs can provide nuanced assessments when generating open-ended answers, they tend to take a stance on no-consensus topics when employed as judges or debaters. These findings underscore the necessity for more sophisticated methods for aligning LLMs without human oversight, highlighting that LLMs cannot fully capture human disagreement even on topics where humans themselves are divided.