LIB-KD: Teaching Inductive Bias for Efficient Vision Transformer Distillation and Compression

TL;DR

This paper presents LIB-KD, a novel ensemble-based distillation method that transfers diverse inductive biases from lightweight teacher models to improve the efficiency and performance of Vision Transformer training and compression.

Contribution

It introduces an ensemble of lightweight teachers with varied inductive biases and a precomputed logits strategy to accelerate and enhance Vision Transformer distillation.

Findings

Ensemble of diverse teachers improves student ViT performance.

Precomputing logits accelerates the distillation process.

Method reduces computational costs while maintaining high accuracy.

Abstract

With the rapid development of computer vision, Vision Transformers (ViTs) offer the tantalising prospect of unified information processing across visual and textual domains due to the lack of inherent inductive biases in ViTs. ViTs require enormous datasets for training. We introduce an innovative ensemble-based distillation approach that distils inductive bias from complementary lightweight teacher models to make their applications practical. Prior systems relied solely on convolution-based teaching. However, this method incorporates an ensemble of light teachers with different architectural tendencies, such as convolution and involution, to jointly instruct the student transformer. Because of these unique inductive biases, instructors can accumulate a wide range of knowledge, even from readily identifiable stored datasets, which leads to enhanced student performance. Our proposed framework LIB-KD also involves precomputing and keeping logits in advance, essentially the unnormalized predictions of the model. This optimisation can accelerate the distillation process by eliminating the need for repeated forward passes during knowledge distillation, significantly reducing the computational burden and enhancing efficiency.