Unlocking In-Context Learning for Natural Datasets Beyond Language Modelling

TL;DR

This paper investigates the properties enabling In-Context Learning in Large Language Models across different modalities, revealing key factors like token repetitions and task difficulty that enhance ICL performance.

Contribution

It systematically uncovers properties supporting ICL emergence in autoregressive models and extends ICL capabilities to visual and EEG datasets.

Findings

Token repetitions in training data improve ICL stability.

Training task difficulty influences ICL emergence.

ICL can be unlocked for visual and EEG datasets.

Abstract

Large Language Models (LLMs) exhibit In-Context Learning (ICL), which enables the model to perform new tasks conditioning only on the examples provided in the context without updating the model's weights. While ICL offers fast adaptation across natural language tasks and domains, its emergence is less straightforward for modalities beyond text. In this work, we systematically uncover properties present in LLMs that support the emergence of ICL for autoregressive models and various modalities by promoting the learning of the needed mechanisms for ICL. We identify exact token repetitions in the training data sequences as an important factor for ICL. Such repetitions further improve stability and reduce transiency in ICL performance. Moreover, we emphasise the significance of training task difficulty for the emergence of ICL. Finally, by applying our novel insights on ICL emergence, we unlock ICL capabilities for various visual datasets and a more challenging EEG classification task.