Positional Bias in Binary Question Answering: How Uncertainty Shapes Model Preferences

TL;DR

This paper investigates how positional bias in binary question answering models varies with answer uncertainty, showing bias is minimal under certainty but increases exponentially as uncertainty grows, across multiple datasets and models.

Contribution

It introduces a systematic analysis of positional bias under varying uncertainty levels and adapts datasets to measure bias in large language models.

Findings

Positional bias is negligible under low uncertainty.

Bias increases exponentially with higher uncertainty.

Models show systematic preference shifts based on option position.

Abstract

Positional bias in binary question answering occurs when a model systematically favors one choice over another based solely on the ordering of presented options. In this study, we quantify and analyze positional bias across five large language models under varying degrees of answer uncertainty. We re-adapted the SQuAD-it dataset by adding an extra incorrect answer option and then created multiple versions with progressively less context and more out-of-context answers, yielding datasets that range from low to high uncertainty. Additionally, we evaluate two naturally higher-uncertainty benchmarks: (1) WebGPT - question pairs with unequal human-assigned quality scores, and (2) Winning Arguments - where models predict the more persuasive argument in Reddit's r/ChangeMyView exchanges. Across each dataset, the order of the "correct" (or higher-quality/persuasive) option is systematically flipped (first placed in position 1, then in position 2) to compute both Preference Fairness and Position Consistency. We observe that positional bias is nearly absent under low-uncertainty conditions, but grows exponentially when it becomes doubtful to decide which option is correct.