Competitive Distillation: A Simple Learning Strategy for Improving Visual Classification

TL;DR

This paper introduces a competitive distillation strategy for training neural networks, where networks compete and learn from each other dynamically, leading to improved visual classification performance across various datasets.

Contribution

The paper proposes a novel competitive distillation approach that allows networks to act as teachers based on performance, enhancing learning through competition and stochastic perturbation.

Findings

Achieves improved accuracy on multiple datasets

Enhances training efficiency compared to traditional methods

Demonstrates robustness across diverse tasks

Abstract

Deep Neural Networks (DNNs) have significantly advanced the field of computer vision. To improve DNN training process, knowledge distillation methods demonstrate their effectiveness in accelerating network training by introducing a fixed learning direction from the teacher network to student networks. In this context, several distillation-based optimization strategies are proposed, e.g., deep mutual learning and self-distillation, as an attempt to achieve generic training performance enhancement through the cooperative training of multiple networks. However, such strategies achieve limited improvements due to the poor understanding of the impact of learning directions among networks across different iterations. In this paper, we propose a novel competitive distillation strategy that allows each network in a group to potentially act as a teacher based on its performance, enhancing the overall learning performance. Competitive distillation organizes a group of networks to perform a shared task and engage in competition, where competitive optimization is proposed to improve the parameter updating process. We further introduce stochastic perturbation in competitive distillation, aiming to motivate networks to induce mutations to achieve better visual representations and global optimum. The experimental results show that competitive distillation achieves promising performance in diverse tasks and datasets.