Contrastive Deep Supervision

TL;DR

This paper introduces Contrastive Deep Supervision, a novel training framework that enhances intermediate layer supervision in deep neural networks using augmentation-based contrastive learning, improving performance across various tasks.

Contribution

It proposes a new contrastive supervision method for intermediate layers, addressing conflicts in traditional deep supervision and enhancing learning in shallow layers.

Findings

Improves image classification accuracy on nine datasets.

Enhances object detection performance.

Effective in supervised, semi-supervised, and knowledge distillation settings.

Abstract

The success of deep learning is usually accompanied by the growth in neural network depth. However, the traditional training method only supervises the neural network at its last layer and propagates the supervision layer-by-layer, which leads to hardship in optimizing the intermediate layers. Recently, deep supervision has been proposed to add auxiliary classifiers to the intermediate layers of deep neural networks. By optimizing these auxiliary classifiers with the supervised task loss, the supervision can be applied to the shallow layers directly. However, deep supervision conflicts with the well-known observation that the shallow layers learn low-level features instead of task-biased high-level semantic features. To address this issue, this paper proposes a novel training framework named Contrastive Deep Supervision, which supervises the intermediate layers with augmentation-based contrastive learning. Experimental results on nine popular datasets with eleven models demonstrate its effects on general image classification, fine-grained image classification and object detection in supervised learning, semi-supervised learning and knowledge distillation. Codes have been released in Github.