How Few-Shot Examples Add Up: A Causal Decomposition of Function Vectors in In-Context Learning

TL;DR

This paper provides a mechanistic, causal explanation of how few-shot prompts influence in-context learning by decomposing function vectors into additive, context-dependent components, highlighting the roles of attention and representation updates.

Contribution

It introduces a causal decomposition framework that explains how few-shot examples shape model behavior through additive and contextualized function vectors, unifying superposition and attention reweighting.

Findings

Function vectors are well-approximated by linear combinations of example sub-vectors.

Models reweight demonstrations based on informativeness and ambiguity, affecting the function vector.

Query-Key alignment primarily drives the quality of the function vector in ambiguous settings.

Abstract

In-context learning (ICL) excels at new tasks from minimal examples, yet we still lack a mechanistic explanation of how few-shot prompts shape a model's function vector (FV)--a causal activation direction that drives task behavior on the ICL query. Across tasks and models, an $n$-shot FV is well-approximated by a linear combination of example-level sub-FVs, suggesting additive and composable contributions from individual demonstrations. Beyond additivity, we show that models contextualize individual examples' representations based on prior examples to adaptively reweight which demonstrations dominate the FV: attention shifts toward examples that are more informative and less ambiguous under the context. Finally, a causal decomposition separates Query-Key routing from Value updates, finding that contextualization's most consistent contributions to FV quality arise from Query-Key alignment--particularly in ambiguous settings--while Value-mediated effects are more heterogeneous. Together, these results unify additive superposition with context-dependent attention reweighting into a mechanistic, testable account of how few-shot prompts implement tasks.