Dynamical Systems Analysis Reveals Functional Regimes in Large Language Models

TL;DR

This paper introduces a neuroscience-inspired dynamical metric to analyze the internal temporal organization of large language models, revealing distinct functional regimes during different tasks.

Contribution

It adapts concepts from neuroscience to transformer models, providing a new method to quantify and differentiate internal dynamical states in large language models.

Findings

Structured reasoning shows higher dynamical metric than noisy regimes.

The metric reliably distinguishes different functional regimes.

Results are consistent across model layers and perturbations.

Abstract

Large language models perform text generation through high-dimensional internal dynamics, yet the temporal organisation of these dynamics remains poorly understood. Most interpretability approaches emphasise static representations or causal interventions, leaving temporal structure largely unexplored. Drawing on neuroscience, where temporal integration and metastability are core markers of neural organisation, we adapt these concepts to transformer models and discuss a composite dynamical metric, computed from activation time-series during autoregressive generation. We evaluate this metric in GPT-2-medium across five conditions: structured reasoning, forced repetition, high-temperature noisy sampling, attention-head pruning, and weight-noise injection. Structured reasoning consistently exhibits elevated metric relative to repetitive, noisy, and perturbed regimes, with statistically significant differences confirmed by one-way ANOVA and large effect sizes in key comparisons. These results are robust to layer selection, channel subsampling, and random seeds. Our findings demonstrate that neuroscience-inspired dynamical metrics can reliably characterise differences in computational organisation across functional regimes in large language models. We stress that the proposed metric captures formal dynamical properties and does not imply subjective experience.