Order Matters: Rethinking Prompt Construction in In-Context Learning

TL;DR

This paper demonstrates that the order of examples in in-context learning prompts significantly impacts model performance, comparable to the effect of example selection, and proposes methods to identify strong orderings using development sets.

Contribution

It challenges the assumption that example selection outweighs ordering effects in ICL and systematically compares their impacts across multiple models and tasks.

Findings

Order effects are comparable to selection effects in performance variance.

Strong orderings can be identified using only development sets.

Prompt design should consider both example selection and ordering.

Abstract

In-context learning (ICL) enables large language models to perform new tasks by conditioning on a sequence of examples. Most prior work reasonably and intuitively assumes that which examples are chosen has a far greater effect on performance than how those examples are ordered, leading to a focus on example selection. We revisit this assumption and conduct a systematic comparison between the effect of selection and ordering. Through controlled experiments on both classification and generation tasks, using multiple open-source model families (0.5B to 27B parameters) and GPT-5, we find that the variance in performance due to different example orderings is comparable to that from using entirely different example sets. Furthermore, we show that strong orderings can be identified using only a development set, achieving performance close to an oracle that selects the best ordering based on test labels. Our findings highlight the equal and intertwined importance of example selection and ordering in prompt design, calling for a reexamination of the assumptions held in ICL.