UNIC: Universal Classification Models via Multi-teacher Distillation

TL;DR

This paper introduces UNIC, a universal classification model learned through multi-teacher distillation, which generalizes across tasks and improves performance by leveraging multiple pretrained models with novel architectural and regularization techniques.

Contribution

It proposes a new multi-teacher distillation framework with architectural enhancements and regularization, enabling a single encoder to perform well across diverse classification tasks.

Findings

Achieves competitive or superior performance compared to individual teachers.

Enhances distillation with a ladder of expendable projectors and teacher dropping.

Demonstrates strong generalization across multiple classification benchmarks.

Abstract

Pretrained models have become a commodity and offer strong results on a broad range of tasks. In this work, we focus on classification and seek to learn a unique encoder able to take from several complementary pretrained models. We aim at even stronger generalization across a variety of classification tasks. We propose to learn such an encoder via multi-teacher distillation. We first thoroughly analyse standard distillation when driven by multiple strong teachers with complementary strengths. Guided by this analysis, we gradually propose improvements to the basic distillation setup. Among those, we enrich the architecture of the encoder with a ladder of expendable projectors, which increases the impact of intermediate features during distillation, and we introduce teacher dropping, a regularization mechanism that better balances the teachers' influence. Our final distillation strategy leads to student models of the same capacity as any of the teachers, while retaining or improving upon the performance of the best teacher for each task. Project page and code: https://europe.naverlabs.com/unic