Self-Knowledge Distillation via Dropout

TL;DR

This paper introduces SD-Dropout, a simple self-knowledge distillation technique using dropout that improves neural network generalization, calibration, robustness, and out-of-distribution detection without extra parameters or data dependence.

Contribution

The paper proposes a novel, parameter-free self-knowledge distillation method using dropout, applicable across various vision tasks, with theoretical and empirical validation.

Findings

Improves generalization across vision tasks.

Enhances calibration, robustness, and out-of-distribution detection.

Does not require additional trainable modules or data.

Abstract

To boost the performance, deep neural networks require deeper or wider network structures that involve massive computational and memory costs. To alleviate this issue, the self-knowledge distillation method regularizes the model by distilling the internal knowledge of the model itself. Conventional self-knowledge distillation methods require additional trainable parameters or are dependent on the data. In this paper, we propose a simple and effective self-knowledge distillation method using a dropout (SD-Dropout). SD-Dropout distills the posterior distributions of multiple models through a dropout sampling. Our method does not require any additional trainable modules, does not rely on data, and requires only simple operations. Furthermore, this simple method can be easily combined with various self-knowledge distillation approaches. We provide a theoretical and experimental analysis of the effect of forward and reverse KL-divergences in our work. Extensive experiments on various vision tasks, i.e., image classification, object detection, and distribution shift, demonstrate that the proposed method can effectively improve the generalization of a single network. Further experiments show that the proposed method also improves calibration performance, adversarial robustness, and out-of-distribution detection ability.