In-Context Function Learning in Large Language Models

TL;DR

This paper investigates how large language models learn in-context functions by comparing their performance to Gaussian Process models, analyzing their biases, and exploring methods to improve their sample efficiency through fine-tuning.

Contribution

It introduces a GP-based framework to quantify LLM in-context learning, analyzes their inductive biases, and demonstrates how fine-tuning can shift these biases for better function learning.

Findings

LLMs' in-context learning curves approach GP lower bounds with more demonstrations.

Predictions are biased towards less smooth GP kernels.

Fine-tuning can shift inductive biases towards smoother functions.

Abstract

Large language models (LLMs) can learn from a few demonstrations provided at inference time. We study this in-context learning phenomenon through the lens of Gaussian Processes (GPs). We build controlled experiments where models observe sequences of multivariate scalar-valued function samples drawn from known GP priors. We evaluate prediction error in relation to the number of demonstrations and compare against two principled references: (i) an empirical GP-regression learner that gives a lower bound on achievable error, and (ii) the expected error of a 1-nearest-neighbor (1-NN) rule, which gives a data-driven upper bound. Across model sizes, we find that LLM learning curves are strongly influenced by the function-generating kernels and approach the GP lower bound as the number of demonstrations increases. We then study the inductive biases of these models using a likelihood-based analysis. We find that LLM predictions are most likely under less smooth GP kernels. Finally, we explore whether post-training can shift these inductive biases and improve sample-efficiency on functions sampled from GPs with smoother kernels. We find that both reinforcement learning and supervised fine-tuning can effectively shift inductive biases in the direction of the training data. Together, our framework quantifies the extent to which LLMs behave like GP learners and provides tools for steering their inductive biases for continuous function learning tasks.