MTA: Multi-Granular Trajectory Alignment for Large Language Model Distillation

TL;DR

This paper introduces Multi-Granular Trajectory Alignment (MTA), a novel method for LLM knowledge distillation that aligns representations across layers and semantic levels to improve transfer quality.

Contribution

MTA is the first approach to align teacher and student models along their entire layer-wise transformation trajectories at multiple semantic granularities.

Findings

MTA outperforms state-of-the-art baselines on standard benchmarks.

Layer-wise and semantic-level alignment improves knowledge transfer.

Ablation studies confirm the effectiveness of each component.

Abstract

Knowledge distillation is a key technique for compressing large language models (LLMs), but most existing methods align representations at fixed layers or token-level outputs, ignoring how representations evolve across depth. As a result, the student is only weakly guided to capture the teacher's internal relational structure during distillation, which limits knowledge transfer. To address this limitation, we propose Multi-Granular Trajectory Alignment (MTA), a framework that aligns teacher and student representations along their layer-wise transformation trajectory. MTA adopts a layer-adaptive strategy: lower layers are aligned at the word level to preserve lexical information, while higher layers operate on phrase-level spans (e.g., noun and verb phrases) to capture compositional semantics. We instantiate this idea through a Dynamic Structural Alignment loss that matches the relative geometry among semantic units within each layer. This design is motivated by empirical findings that Transformer representations become increasingly abstract with depth, and is also consistent with linguistic views in which higher-level meaning emerges through the composition of lower-level lexical units. We further incorporate a Hidden Representation Alignment loss to directly align selected teacher-student layers. Experiments show that MTA consistently outperforms state-of-the-art baselines on standard benchmarks, with ablations confirming the contribution of each component.