Relational Representation Distillation

TL;DR

This paper introduces a novel knowledge distillation method that preserves relational structures in internal representations, outperforming existing methods by better capturing the relationships between instances.

Contribution

It proposes a new objective that maintains relative relationships between instances using separate temperature parameters, bridging contrastive learning and KL divergence.

Findings

Outperforms existing distillation methods across various tasks.

Achieves better alignment with teacher models.

Sometimes surpasses teacher network performance.

Abstract

Knowledge distillation involves transferring knowledge from large, cumbersome teacher models to more compact student models. The standard approach minimizes the Kullback-Leibler (KL) divergence between the probabilistic outputs of a teacher and student network. However, this approach fails to capture important structural relationships in the teacher's internal representations. Recent advances have turned to contrastive learning objectives, but these methods impose overly strict constraints through instance-discrimination, forcing apart semantically similar samples even when they should maintain similarity. This motivates an alternative objective by which we preserve relative relationships between instances. Our method employs separate temperature parameters for teacher and student distributions, with sharper student outputs, enabling precise learning of primary relationships while preserving secondary similarities. We show theoretical connections between our objective and both InfoNCE loss and KL divergence. Experiments demonstrate that our method significantly outperforms existing knowledge distillation methods across diverse knowledge transfer tasks, achieving better alignment with teacher models, and sometimes even outperforms the teacher network.