A Survey to Recent Progress Towards Understanding In-Context Learning

TL;DR

This survey reviews recent progress in understanding In-Context Learning (ICL) in large language models, proposing a data generation perspective to clarify its mechanisms and identify future research directions.

Contribution

It introduces a systematic data generation perspective to reinterpret ICL, defining skill recognition and skill learning, and analyzes their strengths, weaknesses, and commonalities.

Findings

Provides a conceptual framework for ICL mechanisms.

Highlights the potential of data generation perspective.

Suggests future research directions in ICL.

Abstract

In-Context Learning (ICL) empowers Large Language Models (LLMs) with the ability to learn from a few examples provided in the prompt, enabling downstream generalization without the requirement for gradient updates. Despite encouragingly empirical success, the underlying mechanism of ICL remains unclear. Existing research remains ambiguous with various viewpoints, utilizing intuition-driven and ad-hoc technical solutions to interpret ICL. In this paper, we leverage a data generation perspective to reinterpret recent efforts from a systematic angle, demonstrating the potential broader usage of these popular technical solutions. For a conceptual definition, we rigorously adopt the terms of skill recognition and skill learning. Skill recognition selects one learned data generation function previously seen during pre-training while skill learning can learn new data generation functions from in-context data. Furthermore, we provide insights into the strengths and weaknesses of both abilities, emphasizing their commonalities through the perspective of data generation. This analysis suggests potential directions for future research.