PLR: Plackett-Luce for Reordering In-Context Learning Examples

TL;DR

PLR introduces a probabilistic method using the Plackett-Luce model to efficiently find optimal in-context example orderings, improving language model performance on classification and reasoning tasks.

Contribution

It replaces exhaustive ordering search with a learned probability distribution over orderings, enhancing few-shot learning accuracy.

Findings

Consistently improves accuracy on multiple classification benchmarks.

Demonstrates effectiveness on mathematical reasoning tasks.

Uses Gumbel perturb-and-sort for efficient sampling.

Abstract

In-context learning (ICL) adapts large language models by conditioning on a small set of ICL examples, avoiding costly parameter updates. Among other factors, performance is often highly sensitive to the ordering of the examples. However, exhaustive search over the $n!$ possible orderings is infeasible. Therefore more efficient ordering methods use model confidence measures (e.g., label-probability entropy) over label sets or take a direct approach to finding the best ordering. We propose PLR, a probabilistic approach to in-context example ordering that replaces discrete ordering search with learning a probability distribution over orderings with the Plackett-Luce model. PLR models orderings using a Plackett-Luce distribution and iteratively updates its parameters to concentrate probability mass on high-performing orderings under a task-level metric. Candidate orderings are sampled efficiently via a Gumbel perturb-and-sort procedure. Experiments on multiple classification benchmarks show that PLR consistently improves few-shot accuracy for $k \in \{4, 8, 16, 32\}$ examples, and we further demonstrate gains on mathematical reasoning tasks where label-based ordering methods are not applicable. Our code is available at https://github.com/Batorskq/PLR.