Selective Cross-Task Distillation

TL;DR

This paper introduces a selective cross-task distillation method that efficiently identifies and utilizes the most relevant pre-trained teachers across different tasks by bridging label spaces with optimal transport, improving knowledge reuse.

Contribution

It proposes a novel framework for selecting and reusing teachers from diverse tasks via optimal transport, addressing efficiency and semantic gap challenges in knowledge distillation.

Findings

Effective teacher assessment without enumerating all models

Bridging label spaces improves cross-task knowledge transfer

Demonstrated superior performance on benchmark tasks

Abstract

The outpouring of various pre-trained models empowers knowledge distillation by providing abundant teacher resources, but there lacks a developed mechanism to utilize these teachers adequately. With a massive model repository composed of teachers pre-trained on diverse tasks, we must surmount two obstacles when using knowledge distillation to learn a new task. First, given a fixed computing budget, it is not affordable to try each teacher and train the student repeatedly, making it necessary to seek out the most contributive teacher precisely and efficiently. Second, semantic gaps exist between the teachers and the target student since they are trained on different tasks. Thus, we need to extract knowledge from a general label space that may be different from the student's. Faced with these two challenges, we study a new setting named selective cross-task distillation that includes teacher assessment and generalized knowledge reuse. We bridge the teacher's label space and the student's label space through optimal transport. The transportation cost from the teacher's prediction to the student's prediction measures the relatedness between two tasks and acts as an objective for distillation. Our method reuses cross-task knowledge from a distinct label space and efficiently assesses teachers without enumerating the model repository. Experiments demonstrate the effectiveness of our proposed method.