Self-Consistency Falls Short! The Adverse Effects of Positional Bias on Long-Context Problems

TL;DR

Self-consistency, while beneficial for short tasks, negatively impacts long-context problem performance due to persistent position bias, especially in smaller models and longer contexts, revealing limitations in current LLMs.

Contribution

This paper demonstrates that self-consistency degrades long-context task performance and identifies position bias as the key factor, challenging assumptions about its universal effectiveness.

Findings

SC degrades performance on long-context tasks

Position bias worsens with longer contexts and smaller models

SC amplifies positional errors in long contexts

Abstract

Self-consistency (SC) improves the performance of large language models (LLMs) across various tasks and domains that involve short content. However, does this support its effectiveness for long-context problems? We challenge the assumption that SC's benefits generalize to long-context settings, where LLMs often struggle with position bias, the systematic over-reliance on specific context regions-which hinders their ability to utilize information effectively from all parts of their context. Through comprehensive experimentation with varying state-of-the-art models, tasks, and SC formulations, we find that SC not only fails to improve but actively degrades performance on long-context tasks. This degradation is driven by persistent position bias, which worsens with longer context lengths and smaller model sizes but remains invariant to prompt format or task type. Unlike short-context tasks, where SC diversifies reasoning paths, long-context SC amplifies positional errors. These comprehensive results provide valuable insight into the limitations of current LLMs in long-context understanding and highlight the need for more sophisticated approaches.