Benchmarking General-Purpose In-Context Learning

TL;DR

This paper introduces benchmarks for evaluating general-purpose in-context learning (GPICL), demonstrating that task diversity improves generalization but may reduce zero-shot performance, and highlighting the importance of context and memory over scale.

Contribution

We propose two lightweight benchmarks for GPICL, enabling evaluation across diverse tasks and long-horizon interactions, and analyze factors affecting GPICL performance.

Findings

Task diversity enhances generalization in ICL

Increased task diversity can reduce zero-shot capabilities

Model scale alone is not the key to effective ICL

Abstract

In-context learning (ICL) empowers generative models to address new tasks effectively and efficiently on the fly, without relying on any artificially crafted optimization techniques. In this paper, we study extending ICL to address a broader range of tasks with an extended learning horizon and higher improvement potential, namely General Purpose In-Context Learning (GPICL). To this end, we introduce two lightweight benchmarks specifically crafted to train and evaluate GPICL functionalities. Each benchmark encompasses a vast number of tasks characterized by significant task variance. These tasks are also crafted to promote long-horizon in-context learning through continuous generation and interaction, covering domains such as language modeling, decision-making, and world modeling. The benchmarks necessitate the models to leverage contexts and history interactions to enhance their capabilities, which we believe to be the key characteristics of GPICL. Our experiments indicate that the diversity of training tasks is positively correlated with the ability to generalize with ICL, but inversely correlated with zero-shot capabilities. Additionally, our findings indicate that the scale of parameters alone may not be crucial for ICL or GPICL, suggesting alternative approaches such as increasing the scale of contexts and memory states.