Multi-Level Feature Distillation of Joint Teachers Trained on Distinct Image Datasets

TL;DR

This paper introduces a multi-level feature distillation method from multiple teachers trained on different datasets, improving student model performance across various benchmarks by combining features at multiple levels.

Contribution

The paper presents a novel multi-level feature distillation framework that effectively combines teachers trained on distinct datasets, surpassing traditional training methods.

Findings

Significant performance improvements over single-dataset training.

Effective combination of teachers with different architectures.

Validated on multiple image classification and action recognition benchmarks.

Abstract

We propose a novel teacher-student framework to distill knowledge from multiple teachers trained on distinct datasets. Each teacher is first trained from scratch on its own dataset. Then, the teachers are combined into a joint architecture, which fuses the features of all teachers at multiple representation levels. The joint teacher architecture is fine-tuned on samples from all datasets, thus gathering useful generic information from all data samples. Finally, we employ a multi-level feature distillation procedure to transfer the knowledge to a student model for each of the considered datasets. We conduct image classification experiments on seven benchmarks, and action recognition experiments on three benchmarks. To illustrate the power of our feature distillation procedure, the student architectures are chosen to be identical to those of the individual teachers. To demonstrate the flexibility of our approach, we combine teachers with distinct architectures. We show that our novel Multi-Level Feature Distillation (MLFD) can significantly surpass equivalent architectures that are either trained on individual datasets, or jointly trained on all datasets at once. Furthermore, we confirm that each step of the proposed training procedure is well motivated by a comprehensive ablation study. We publicly release our code at https://github.com/AdrianIordache/MLFD.