Cross-Architecture Distillation Made Simple with Redundancy Suppression

TL;DR

This paper introduces a simple, efficient method for cross-architecture knowledge distillation that suppresses redundant information, outperforming complex existing methods like OFA with fewer parameters and broad applicability.

Contribution

The authors propose a novel redundancy suppression distillation (RSD) loss that simplifies cross-architecture knowledge transfer without architecture-specific modules.

Findings

Outperforms OFA on CIFAR-100 and ImageNet-1k benchmarks

Uses fewer parameters than existing methods

Provides a simple, effective baseline for cross-architecture distillation

Abstract

We describe a simple method for cross-architecture knowledge distillation, where the knowledge transfer is cast into a redundant information suppression formulation. Existing methods introduce sophisticated modules, architecture-tailored designs, and excessive parameters, which impair their efficiency and applicability. We propose to extract the architecture-agnostic knowledge in heterogeneous representations by reducing the redundant architecture-exclusive information. To this end, we present a simple redundancy suppression distillation (RSD) loss, which comprises cross-architecture invariance maximisation and feature decorrelation objectives. To prevent the student from entirely losing its architecture-specific capabilities, we further design a lightweight module that decouples the RSD objective from the student's internal representations. Our method is devoid of the architecture-specific designs and complex operations in the pioneering method of OFA. It outperforms OFA on CIFAR-100 and ImageNet-1k benchmarks with only a fraction of their parameter overhead, which highlights its potential as a simple and strong baseline to the cross-architecture distillation community.