Context-Parametric Inversion: Why Instruction Finetuning Can Worsen Context Reliance

TL;DR

This paper investigates why instruction finetuning can unexpectedly reduce a language model's reliance on input context during knowledge conflicts, revealing a phenomenon called context-parametric inversion across multiple models and datasets.

Contribution

The study uncovers the counterintuitive effect of instruction finetuning decreasing context reliance in knowledge conflicts and analyzes its causes through controlled experiments and theoretical insights.

Findings

Context reliance initially increases then decreases during finetuning.

The phenomenon occurs across multiple datasets and model families.

Mitigation strategies offer limited but valuable improvements.

Abstract

A standard practice when using large language models is for users to supplement their instruction with an input context containing new information for the model to process. However, models struggle to reliably follow the input context, especially when it conflicts with their parametric knowledge from pretraining. In-principle, one would expect models to adapt to the user context better after instruction finetuning, particularly when handling knowledge conflicts. However, we observe a surprising failure mode: during instruction tuning, the context reliance under knowledge conflicts initially increases as expected, but then gradually decreases as instruction finetuning progresses. This happens while the performance on standard benchmarks keeps on increasing far after this drop. We call this phenomenon context-parametric inversion and observe it across multiple general purpose instruction tuning datasets such as TULU, Alpaca and Ultrachat, across different model families like Llama, Mistral, and Pythia. We perform various controlled studies and theoretical analysis to show that context-parametric inversion occurs due to examples in the instruction finetuning data where the input context provides information that aligns with model's parametric knowledge. Our analysis suggests some natural mitigation strategies with limited but insightful gains, and serves as a useful starting point in addressing this deficiency in instruction finetuning.