Rethinking Invariance in In-context Learning

TL;DR

This paper introduces InvICL, a new method for in-context learning that achieves permutation invariance while maintaining high performance, leading to better generalization across different input lengths.

Contribution

The paper identifies key elements for invariant ICL and proposes InvICL, which outperforms existing methods in benchmarks by balancing invariance and information retention.

Findings

InvICL surpasses previous models in benchmark datasets.

InvICL demonstrates superior generalization across input lengths.

Existing invariant methods often trade off performance for invariance.

Abstract

In-Context Learning (ICL) has emerged as a pivotal capability of auto-regressive large language models, yet it is hindered by a notable sensitivity to the ordering of context examples regardless of their mutual independence. To address this issue, recent studies have introduced several variant algorithms of ICL that achieve permutation invariance. However, many of these do not exhibit comparable performance with the standard auto-regressive ICL algorithm. In this work, we identify two crucial elements in the design of an invariant ICL algorithm: information non-leakage and context interdependence, which are not simultaneously achieved by any of the existing methods. These investigations lead us to the proposed Invariant ICL (InvICL), a methodology designed to achieve invariance in ICL while ensuring the two properties. Empirically, our findings reveal that InvICL surpasses previous models, both invariant and non-invariant, in most benchmark datasets, showcasing superior generalization capabilities across varying input lengths. Code is available at https://github.com/PKU-ML/InvICL.