TRACE for Tracking the Emergence of Semantic Representations in Transformers

TL;DR

This paper introduces TRACE, a diagnostic framework that tracks the emergence of linguistic abstractions in transformer models by analyzing geometric, informational, and linguistic signals during training, revealing phase transitions linked to abstraction development.

Contribution

The paper presents TRACE, a novel method combining multiple signals and a synthetic data generator to study how linguistic structures emerge in transformers over training.

Findings

Phase transitions align with curvature collapse and dimension stabilization.

Emerging syntactic and semantic accuracy coincides with geometric shifts.

Abstraction patterns are consistent across different model architectures.

Abstract

Modern transformer models exhibit phase transitions during training, distinct shifts from memorisation to abstraction, but the mechanisms underlying these transitions remain poorly understood. Prior work has often focused on endpoint representations or isolated signals like curvature or mutual information, typically in symbolic or arithmetic domains, overlooking the emergence of linguistic structure. We introduce TRACE (Tracking Representation Abstraction and Compositional Emergence), a diagnostic framework combining geometric, informational, and linguistic signals to detect phase transitions in Transformer-based LMs. TRACE leverages a frame-semantic data generation method, ABSynth, that produces annotated synthetic corpora with controllable complexity, lexical distributions, and structural entropy, while being fully annotated with linguistic categories, enabling precise analysis of abstraction emergence. Experiments reveal that (i) phase transitions align with clear intersections between curvature collapse and dimension stabilisation; (ii) these geometric shifts coincide with emerging syntactic and semantic accuracy; (iii) abstraction patterns persist across architectural variants, with components like feedforward networks affecting optimisation stability rather than fundamentally altering trajectories. This work advances our understanding of how linguistic abstractions emerge in LMs, offering insights into model interpretability, training efficiency, and compositional generalisation that could inform more principled approaches to LM development.