Relative Difficulty Distillation for Semantic Segmentation

TL;DR

This paper introduces Relative Difficulty Distillation (RDD), a novel pixel-level knowledge distillation method for semantic segmentation that leverages the relative learning difficulty between teacher and student networks to improve training stability and performance.

Contribution

It redefines knowledge transfer based on relative sample difficulty and proposes a two-stage framework that avoids multiple loss optimization, enhancing distillation effectiveness.

Findings

RDD outperforms state-of-the-art KD methods on multiple datasets.

RDD can be combined with existing KD methods for improved results.

Extensive experiments validate RDD's effectiveness and stability.

Abstract

Current knowledge distillation (KD) methods primarily focus on transferring various structured knowledge and designing corresponding optimization goals to encourage the student network to imitate the output of the teacher network. However, introducing too many additional optimization objectives may lead to unstable training, such as gradient conflicts. Moreover, these methods ignored the guidelines of relative learning difficulty between the teacher and student networks. Inspired by human cognitive science, in this paper, we redefine knowledge from a new perspective -- the student and teacher networks' relative difficulty of samples, and propose a pixel-level KD paradigm for semantic segmentation named Relative Difficulty Distillation (RDD). We propose a two-stage RDD framework: Teacher-Full Evaluated RDD (TFE-RDD) and Teacher-Student Evaluated RDD (TSE-RDD). RDD allows the teacher network to provide effective guidance on learning focus without additional optimization goals, thus avoiding adjusting learning weights for multiple losses. Extensive experimental evaluations using a general distillation loss function on popular datasets such as Cityscapes, CamVid, Pascal VOC, and ADE20k demonstrate the effectiveness of RDD against state-of-the-art KD methods. Additionally, our research showcases that RDD can integrate with existing KD methods to improve their upper performance bound.