LLMs learn governing principles of dynamical systems, revealing an in-context neural scaling law

TL;DR

This paper demonstrates that large language models can learn and predict the governing principles of dynamical systems through in-context learning, revealing a neural scaling law related to input context length.

Contribution

It shows that LLMs can extrapolate physical laws from text data and introduces an algorithm for extracting probability densities of multi-digit numbers from models.

Findings

LLaMA 2 predicts dynamical systems accurately without fine-tuning.

Prediction accuracy improves with longer input context, indicating an in-context neural scaling law.

Introduces an efficient method for extracting probability densities from LLMs.

Abstract

Pretrained large language models (LLMs) are surprisingly effective at performing zero-shot tasks, including time-series forecasting. However, understanding the mechanisms behind such capabilities remains highly challenging due to the complexity of the models. We study LLMs' ability to extrapolate the behavior of dynamical systems whose evolution is governed by principles of physical interest. Our results show that LLaMA 2, a language model trained primarily on texts, achieves accurate predictions of dynamical system time series without fine-tuning or prompt engineering. Moreover, the accuracy of the learned physical rules increases with the length of the input context window, revealing an in-context version of neural scaling law. Along the way, we present a flexible and efficient algorithm for extracting probability density functions of multi-digit numbers directly from LLMs.