Contrastive Classification and Representation Learning with Probabilistic Interpretation

TL;DR

This paper introduces a unified objective function that combines contrastive learning and cross entropy to improve neural network training, leading to more robust and stable classification performance.

Contribution

It proposes a novel joint learning approach for classifiers and representations using a contrastive-based objective with probabilistic interpretation, enhancing robustness and stability.

Findings

Significant improvement over standard cross entropy loss.

Enhanced training stability and robustness.

Effective joint learning of classifier and backbone.

Abstract

Cross entropy loss has served as the main objective function for classification-based tasks. Widely deployed for learning neural network classifiers, it shows both effectiveness and a probabilistic interpretation. Recently, after the success of self supervised contrastive representation learning methods, supervised contrastive methods have been proposed to learn representations and have shown superior and more robust performance, compared to solely training with cross entropy loss. However, cross entropy loss is still needed to train the final classification layer. In this work, we investigate the possibility of learning both the representation and the classifier using one objective function that combines the robustness of contrastive learning and the probabilistic interpretation of cross entropy loss. First, we revisit a previously proposed contrastive-based objective function that approximates cross entropy loss and present a simple extension to learn the classifier jointly. Second, we propose a new version of the supervised contrastive training that learns jointly the parameters of the classifier and the backbone of the network. We empirically show that our proposed objective functions show a significant improvement over the standard cross entropy loss with more training stability and robustness in various challenging settings.