Multi-Label Contrastive Learning : A Comprehensive Study

TL;DR

This paper provides a comprehensive analysis of contrastive learning loss functions for multi-label classification, highlighting their effectiveness across diverse datasets and applications in vision and NLP.

Contribution

It offers an in-depth empirical study of contrastive loss in multi-label settings, addressing challenges and demonstrating its strengths in large-label datasets.

Findings

Contrastive learning improves multi-label classification by capturing label interactions.

Robust optimization schemes enhance contrastive loss effectiveness.

Performance varies with label count and dataset size, excelling in large-label scenarios.

Abstract

Multi-label classification, which involves assigning multiple labels to a single input, has emerged as a key area in both research and industry due to its wide-ranging applications. Designing effective loss functions is crucial for optimizing deep neural networks for this task, as they significantly influence model performance and efficiency. Traditional loss functions, which often maximize likelihood under the assumption of label independence, may struggle to capture complex label relationships. Recent research has turned to supervised contrastive learning, a method that aims to create a structured representation space by bringing similar instances closer together and pushing dissimilar ones apart. Although contrastive learning offers a promising approach, applying it to multi-label classification presents unique challenges, particularly in managing label interactions and data structure. In this paper, we conduct an in-depth study of contrastive learning loss for multi-label classification across diverse settings. These include datasets with both small and large numbers of labels, datasets with varying amounts of training data, and applications in both computer vision and natural language processing. Our empirical results indicate that the promising outcomes of contrastive learning are attributable not only to the consideration of label interactions but also to the robust optimization scheme of the contrastive loss. Furthermore, while the supervised contrastive loss function faces challenges with datasets containing a small number of labels and ranking-based metrics, it demonstrates excellent performance, particularly in terms of Macro-F1, on datasets with a large number of labels.