A Measure-Theoretic Analysis of Reasoning: Structural Generalization and Approximation Limits

TL;DR

This paper provides a measure-theoretic framework for understanding reasoning in large language models, highlighting fundamental limits on out-of-distribution generalization related to architecture and depth.

Contribution

It formalizes reasoning via optimal transport, derives bounds on OOD generalization, and establishes depth and architecture constraints for Transformers.

Findings

Shift-invariant mechanisms like Rotary Embeddings improve OOD robustness.

Deeper physical layers are necessary to prevent representation collapse.

Generalization risk increases monotonically with domain shift.

Abstract

While empirical scaling laws for LLM reasoning are well-documented, the theoretical mechanisms governing out-of-distribution (OOD) generalization remain elusive. We formalize reasoning via optimal transport, projecting discrete trajectories into a continuous metric space to quantify domain shifts using the Wasserstein-1 distance. Invoking Kantorovich duality, we bound OOD generalization via architectural Lipschitz continuity and functional approximation limits. This exposes two primary constraints. First, position-dependent attention (e.g., Absolute Positional Encoding) fails to preserve shift invariance, yielding an $\Omega(1)$ Lipschitz constant and expected risk, whereas shift-invariant mechanisms (e.g., Rotary Embeddings) preserve equivariance and bound the error. Second, by mapping sequential backtracking to a Dyck-$k$ language, we establish a strict circuit depth lower bound for $\text{TC}^0$ Transformers. Scaling physical layer depth is necessary to avert representation collapse -- a constraint that scaling representation width cannot bypass due to irreducible approximation bounds in Barron spaces. Evaluations across 54 Transformer configurations on combinatorial search corroborate these bounds, demonstrating that generalization risk degrades monotonically with the Wasserstein domain shift.