Distilled Circuits: A Mechanistic Study of Internal Restructuring in Knowledge Distillation

TL;DR

This paper investigates how internal structures of neural networks change during knowledge distillation, revealing that students reorganize and simplify internal components while maintaining overall function, with implications for robustness.

Contribution

It introduces a mechanistic interpretability approach to analyze internal circuit reorganization during knowledge distillation, including a new influence-weighted component alignment metric.

Findings

Students reorganize and compress internal circuits

Distilled models rely on fewer components

Distillation preserves broad functions but alters internal computation

Abstract

Knowledge distillation compresses a larger neural model (teacher) into smaller, faster student models by training them to match teacher outputs. However, the internal computational transformations that occur during this process remain poorly understood. We apply techniques from mechanistic interpretability to analyze how internal circuits, representations, and activation patterns differ between teachers and students. Focusing on GPT2 and its distilled counterpart DistilGPT2, and generalizing our findings to both bidirectional architectures and larger model pairs, we find that student models can reorganize, compress, and discard teacher components, often resulting in a stronger reliance on fewer individual components. To quantify functional alignment beyond output similarity, we introduce an alignment metric based on influence-weighted component similarity, validated across multiple tasks. Our findings reveal that while knowledge distillation preserves broad functional behaviors, it also causes significant shifts in internal computation, with important implications for the robustness and generalization capacity of distilled models.