Where to show Demos in Your Prompt: A Positional Bias of In-Context Learning

TL;DR

This paper uncovers a positional bias in in-context learning where the placement of demonstrations within prompts significantly impacts model accuracy and stability, with demos at the start generally yielding better results.

Contribution

It systematically studies the positional bias in ICL, introduces metrics to quantify it, and demonstrates its significant effect across multiple tasks and models.

Findings

Demos at prompt start improve accuracy by up to 6 points.

Placing demos at the end can flip over 30% of predictions.

Smaller models are more sensitive to demo placement.

Abstract

In-context learning (ICL) is a critical emerging capability of large language models (LLMs), enabling few-shot learning during inference by including a few demonstrations (demos) in the prompt. However, it has been found that ICL's performance can be sensitive to the choices of demos and their order. This paper investigates an unexplored new positional bias of ICL for the first time: we observe that the predictions and accuracy can drift drastically when the positions of demos, the system prompt, and the user message in LLM input are varied. We refer to this bias as DEMOS' POSITION IN PROMPT (DPP) bias. We design a systematic evaluation pipeline to study this type of positional bias across classification, question answering, summarization, and reasoning tasks. We introduce two metrics, ACCURACY-CHANGE and PREDICTION-CHANGE, to quantify net gains and output volatility induced by changes in the demos' position. Extensive experiments on ten LLMs from four open-source model families (QWEN, LLAMA3, MISTRAL, COHERE) verify that the bias significantly affects their accuracy and predictions: placing demos at the start of the prompt yields the most stable and accurate outputs with gains of up to +6 points. In contrast, placing demos at the end of the user message flips over 30\% of predictions without improving correctness on QA tasks. Smaller models are most affected by this sensitivity, though even large models remain marginally affected on complex tasks.