Feature Representation Transferring to Lightweight Models via Perception Coherence

TL;DR

This paper introduces a novel method for transferring feature representations from large teacher models to lightweight student models by leveraging perception coherence, which focuses on preserving dissimilarity rankings rather than exact geometries.

Contribution

The paper proposes a new perception coherence concept and a corresponding loss function that improves feature transfer by relaxing geometric constraints, extending rankings into a probabilistic framework.

Findings

Outperforms strong baseline methods in feature transfer tasks

Provides a probabilistic perspective on perception coherence

Achieves comparable or better performance than existing methods

Abstract

In this paper, we propose a method for transferring feature representation to lightweight student models from larger teacher models. We mathematically define a new notion called \textit{perception coherence}. Based on this notion, we propose a loss function, which takes into account the dissimilarities between data points in feature space through their ranking. At a high level, by minimizing this loss function, the student model learns to mimic how the teacher model \textit{perceives} inputs. More precisely, our method is motivated by the fact that the representational capacity of the student model is weaker than the teacher model. Hence, we aim to develop a new method allowing for a better relaxation. This means that, the student model does not need to preserve the absolute geometry of the teacher one, while preserving global coherence through dissimilarity ranking. Importantly, while rankings are defined only on finite sets, our notion of \textit{perception coherence} extends them into a probabilistic form. This formulation depends on the input distribution and applies to general dissimilarity metrics. Our theoretical insights provide a probabilistic perspective on the process of feature representation transfer. Our experiments results show that our method outperforms or achieves on-par performance compared to strong baseline methods for representation transferring.