The Order Effect: Investigating Prompt Sensitivity to Input Order in LLMs

TL;DR

This paper examines how the order of input prompts affects the performance of large language models, revealing persistent sensitivity that impacts reliability and highlighting the need for more robust solutions.

Contribution

It provides a systematic analysis of order sensitivity in LLMs across multiple tasks, demonstrating that input order significantly influences outputs despite recent mitigation efforts.

Findings

Input order significantly impacts LLM performance

Few-shot prompting offers limited mitigation

Order sensitivity remains a risk in high-stakes applications

Abstract

As large language models (LLMs) become integral to diverse applications, ensuring their reliability under varying input conditions is crucial. One key issue affecting this reliability is order sensitivity, wherein slight variations in the input arrangement can lead to inconsistent or biased outputs. Although recent advances have reduced this sensitivity, the problem remains unresolved. This paper investigates the extent of order sensitivity in LLMs whose internal components are hidden from users (such as closed-source models or those accessed via API calls). We conduct experiments across multiple tasks, including paraphrasing, relevance judgment, and multiple-choice questions. Our results show that input order significantly affects performance across tasks, with shuffled inputs leading to measurable declines in output accuracy. Few-shot prompting demonstrates mixed effectiveness and offers partial mitigation; however, fails to fully resolve the problem. These findings highlight persistent risks, particularly in high-stakes applications, and point to the need for more robust LLMs or improved input-handling techniques in future development.