Inference and Verbalization Functions During In-Context Learning

TL;DR

This paper investigates how large language models perform in-context learning by separating the inference and verbalization functions, revealing that the inference process is invariant to label remappings and can be localized in specific model layers.

Contribution

The paper proposes and empirically validates the hypothesis that inference and verbalization functions are distinct in LLMs, with inference being invariant to label remappings and localized in certain layers.

Findings

Inference functions are invariant to label remappings.

Inference and verbalization functions can be localized in specific layers.

Models share the same inference function across different label settings.

Abstract

Large language models (LMs) are capable of in-context learning from a few demonstrations (example-label pairs) to solve new tasks during inference. Despite the intuitive importance of high-quality demonstrations, previous work has observed that, in some settings, ICL performance is minimally affected by irrelevant labels (Min et al., 2022). We hypothesize that LMs perform ICL with irrelevant labels via two sequential processes: an inference function that solves the task, followed by a verbalization function that maps the inferred answer to the label space. Importantly, we hypothesize that the inference function is invariant to remappings of the label space (e.g., "true"/"false" to "cat"/"dog"), enabling LMs to share the same inference function across settings with different label words. We empirically validate this hypothesis with controlled layer-wise interchange intervention experiments. Our findings confirm the hypotheses on multiple datasets and tasks (natural language inference, sentiment analysis, and topic classification) and further suggest that the two functions can be localized in specific layers across various open-sourced models, including GEMMA-7B, MISTRAL-7B-V0.3, GEMMA-2-27B, and LLAMA-3.1-70B.