# Similarity-Preserving Knowledge Distillation

## TL;DR

This paper introduces a novel knowledge distillation method that emphasizes preserving pairwise input similarities in the student's activation space, leading to improved transfer of semantic information.

## Contribution

It proposes a similarity-preserving distillation loss that encourages students to maintain input similarity relationships without mimicking teacher representations.

## Key findings

- Effective on multiple datasets
- Outperforms traditional distillation methods
- Preserves semantic input relationships

## Abstract

Knowledge distillation is a widely applicable technique for training a student neural network under the guidance of a trained teacher network. For example, in neural network compression, a high-capacity teacher is distilled to train a compact student; in privileged learning, a teacher trained with privileged data is distilled to train a student without access to that data. The distillation loss determines how a teacher's knowledge is captured and transferred to the student. In this paper, we propose a new form of knowledge distillation loss that is inspired by the observation that semantically similar inputs tend to elicit similar activation patterns in a trained network. Similarity-preserving knowledge distillation guides the training of a student network such that input pairs that produce similar (dissimilar) activations in the teacher network produce similar (dissimilar) activations in the student network. In contrast to previous distillation methods, the student is not required to mimic the representation space of the teacher, but rather to preserve the pairwise similarities in its own representation space. Experiments on three public datasets demonstrate the potential of our approach.