Channel Self-Supervision for Online Knowledge Distillation

TL;DR

This paper introduces CSS, a novel online knowledge distillation method that enhances diversity among peer models through self-supervision and a dual-network structure, leading to improved performance and generalization.

Contribution

CSS innovatively combines self-supervised learning with a dual-network structure to address homogeneity in online distillation, improving diversity and effectiveness.

Findings

Outperforms OKDDip in diversity and accuracy on CIFAR-100.

Achieves state-of-the-art results on fine-grained datasets.

Demonstrates strong generalization across multiple datasets.

Abstract

Recently, researchers have shown an increased interest in the online knowledge distillation. Adopting an one-stage and end-to-end training fashion, online knowledge distillation uses aggregated intermediated predictions of multiple peer models for training. However, the absence of a powerful teacher model may result in the homogeneity problem between group peers, affecting the effectiveness of group distillation adversely. In this paper, we propose a novel online knowledge distillation method, \textbf{C}hannel \textbf{S}elf-\textbf{S}upervision for Online Knowledge Distillation (CSS), which structures diversity in terms of input, target, and network to alleviate the homogenization problem. Specifically, we construct a dual-network multi-branch structure and enhance inter-branch diversity through self-supervised learning, adopting the feature-level transformation and augmenting the corresponding labels. Meanwhile, the dual network structure has a larger space of independent parameters to resist the homogenization problem during distillation. Extensive quantitative experiments on CIFAR-100 illustrate that our method provides greater diversity than OKDDip and we also give pretty performance improvement, even over the state-of-the-art such as PCL. The results on three fine-grained datasets (StanfordDogs, StanfordCars, CUB-200-211) also show the significant generalization capability of our approach.