Hybrid Discriminative-Generative Training via Contrastive Learning

TL;DR

This paper unifies contrastive and supervised learning through hybrid discriminative-generative training of energy-based models, leading to improved classification, robustness, and out-of-distribution detection on CIFAR datasets.

Contribution

It introduces a unified framework connecting contrastive and supervised learning via energy-based models, outperforming existing methods in accuracy and robustness.

Findings

Improved classification accuracy on CIFAR-10 and CIFAR-100.

Enhanced robustness and out-of-distribution detection.

Better calibration of model predictions.

Abstract

Contrastive learning and supervised learning have both seen significant progress and success. However, thus far they have largely been treated as two separate objectives, brought together only by having a shared neural network. In this paper we show that through the perspective of hybrid discriminative-generative training of energy-based models we can make a direct connection between contrastive learning and supervised learning. Beyond presenting this unified view, we show our specific choice of approximation of the energy-based loss outperforms the existing practice in terms of classification accuracy of WideResNet on CIFAR-10 and CIFAR-100. It also leads to improved performance on robustness, out-of-distribution detection, and calibration.