On the generalization of language models from in-context learning and finetuning: a controlled study

TL;DR

This paper compares in-context learning and fine-tuning in large language models, revealing that ICL often generalizes more flexibly and proposing a method to enhance fine-tuning through in-context reasoning traces.

Contribution

The study introduces novel datasets for evaluating generalization, and proposes a method to improve fine-tuning by incorporating in-context reasoning traces.

Findings

ICL can generalize inferences more flexibly than fine-tuning in data-matched settings

Adding in-context reasoning traces to fine-tuning data improves generalization

Fine-tuning can sometimes generalize to reversals within larger knowledge structures

Abstract

Large language models exhibit exciting capabilities, yet can show surprisingly narrow generalization from finetuning. E.g. they can fail to generalize to simple reversals of relations they are trained on, or fail to make simple logical deductions based on trained information. These failures to generalize factual information from fine-tuning can significantly hinder the reasoning capabilities of these models. On the other hand, language models' in-context learning (ICL) shows different inductive biases and deductive reasoning capabilities. Here, we explore these differences in generalization and deductive reasoning between in-context- and fine-tuning-based learning. To do so, we constructed several novel datasets to evaluate and improve models' abilities to make generalizations over factual information from novel data. These datasets are designed to create clean tests of generalization, by isolating the knowledge in the dataset from that in pretraining. We expose pretrained large models to controlled subsets of the information in these datasets -- either through ICL or fine-tuning -- and evaluate their performance on test sets that require various types of generalization. We find overall that in data-matched settings, ICL can generalize several types of inferences more flexibly than fine-tuning (though we also find some qualifications of prior findings, such as cases when fine-tuning can generalize to reversals embedded in a larger structure of knowledge). We build on these findings to propose a method to enable improved generalization from fine-tuning: adding in-context reasoning traces to finetuning data. We show that this method improves generalization across various splits of our datasets and other benchmarks. Our results have implications for understanding the generalization afforded by different modes of learning in language models, and practically improving their performance.