Batch-ICL: Effective, Efficient, and Order-Agnostic In-Context Learning

TL;DR

Batch-ICL introduces an order-agnostic, efficient inference method for in-context learning by aggregating meta-gradients from multiple 1-shot computations, outperforming traditional ICL in accuracy and resource use.

Contribution

The paper proposes Batch-ICL, a novel order-agnostic inference algorithm for ICL that improves performance and efficiency over standard methods.

Findings

Batch-ICL outperforms most permutations of ICL examples.

It can surpass the best order performance of standard ICL.

The method reduces computational resources needed.

Abstract

In this paper, by treating in-context learning (ICL) as a meta-optimization process, we explain why LLMs are sensitive to the order of ICL examples. This understanding leads us to the development of Batch-ICL, an effective, efficient, and order-agnostic inference algorithm for ICL. Differing from the standard N-shot learning approach, Batch-ICL employs $N$ separate 1-shot forward computations and aggregates the resulting meta-gradients. These aggregated meta-gradients are then applied to the forward computation of a zero-shot query to generate the final prediction. This batch processing approach renders the LLM agnostic to the order of ICL examples. Through extensive experiments and analysis, we demonstrate that Batch-ICL consistently outperforms most permutations of ICL examples. In some cases, it even exceeds the performance of the best order for standard ICL, all while reducing the computational resources required. Furthermore, we develop a novel variant of Batch-ICL featuring multiple "epochs" of meta-optimization. This variant implicitly explores permutations of ICL examples, further enhancing ICL performance.