Dynamical Behaviors of the Gradient Flows for In-Context Learning

TL;DR

This paper derives and analyzes the differential equations governing gradient flows in linear in-context learning, providing insights into their geometric structure, invariants, and critical points to better understand training dynamics.

Contribution

It introduces a general framework for the differential equations of gradient flows in linear in-context learning and explores their geometric properties.

Findings

Identification of invariants, optima, and saddle points in gradient flows

Quantitative analysis of gradient flow behavior across parameters and data

Enhanced understanding of training dynamics in linear in-context learning

Abstract

We derive the system of differential equations for the gradient flow characterizing the training process of linear in-context learning in full generality. Next, we explore the geometric structure of the gradient flows in two instances, including identifying its invariants, optimum, and saddle points. This understanding allows us to quantify the behavior of the two gradient flows under the full generality of parameters and data.