One Task Vector is not Enough: A Large-Scale Study for In-Context Learning

TL;DR

This study introduces a large-scale dataset and analysis revealing that in-context learning in LLMs involves multiple task vectors and varies by task complexity and layer, challenging the single-vector assumption.

Contribution

The paper presents QuiteAFew, a large dataset for analyzing in-context learning, and demonstrates that multiple vectors are needed for complex tasks, providing new insights into task representation.

Findings

Task vector performance peaks at intermediate layers.

Effectiveness varies significantly by task type.

Complex tasks rely on multiple, subtask-specific vectors.

Abstract

In-context learning (ICL) enables Large Language Models (LLMs) to adapt to new tasks using few examples, with task vectors - specific hidden state activations - hypothesized to encode task information. Existing studies are limited by small-scale benchmarks, restricting comprehensive analysis. We introduce QuiteAFew, a novel dataset of 3,096 diverse few-shot tasks, each with 30 input-output pairs derived from the Alpaca dataset. Experiments with Llama-3-8B on QuiteAFew reveal: (1) task vector performance peaks at an intermediate layer (e.g., 15th), (2) effectiveness varies significantly by task type, and (3) complex tasks rely on multiple, subtask-specific vectors rather than a single vector, suggesting distributed task knowledge representation.