From Text to Trajectories: GPT-2 as an ODE Solver via In-Context

TL;DR

This paper demonstrates that GPT-2, when prompted appropriately, can effectively solve ordinary differential equations, showing convergence, exponential accuracy improvements, and robust generalization to out-of-distribution problems, revealing insights into in-context learning mechanisms.

Contribution

It introduces a novel approach to use GPT-2 as an ODE solver through in-context learning, highlighting its ability to learn algorithms and generalize beyond training distributions.

Findings

GPT-2 can learn a meta-ODE solving algorithm.

Convergence behavior is comparable or superior to Euler's method.

Exponential accuracy gains with more demonstrations and robust OOD generalization.

Abstract

In-Context Learning (ICL) has emerged as a new paradigm in large language models (LLMs), enabling them to perform novel tasks by conditioning on a few examples embedded in the prompt. Yet, the highly nonlinear behavior of ICL for NLP tasks remains poorly understood. To shed light on its underlying mechanisms, this paper investigates whether LLMs can solve ordinary differential equations (ODEs) under the ICL setting. We formulate standard ODE problems and their solutions as sequential prompts and evaluate GPT-2 models on these tasks. Experiments on two types of ODEs show that GPT-2 can effectively learn a meta-ODE algorithm, with convergence behavior comparable to, or better than, the Euler method, and achieve exponential accuracy gains with increasing numbers of demonstrations. Moreover, the model generalizes to out-of-distribution (OOD) problems, demonstrating robust extrapolation capabilities. These empirical findings provide new insights into the mechanisms of ICL in NLP and its potential for solving nonlinear numerical problems.