Transferability of Representations Learned using Supervised Contrastive Learning Trained on a Multi-Domain Dataset

TL;DR

This paper investigates how supervised contrastive learning trained on a multi-domain dataset improves the transferability of learned representations across diverse downstream tasks, outperforming traditional cross-entropy models.

Contribution

It provides empirical evidence that supervised contrastive learning on multi-domain data enhances transferability of representations compared to cross-entropy training.

Findings

Supervised contrastive learning outperforms cross-entropy by 6.05% on average across 7 datasets.

Models trained with contrastive learning transfer better to new domains.

Contrastive models learn more robust and generalizable representations.

Abstract

Contrastive learning has shown to learn better quality representations than models trained using cross-entropy loss. They also transfer better to downstream datasets from different domains. However, little work has been done to explore the transferability of representations learned using contrastive learning when trained on a multi-domain dataset. In this paper, a study has been conducted using the Supervised Contrastive Learning framework to learn representations from the multi-domain DomainNet dataset and then evaluate the transferability of the representations learned on other downstream datasets. The fixed feature linear evaluation protocol will be used to evaluate the transferability on 7 downstream datasets that were chosen across different domains. The results obtained are compared to a baseline model that was trained using the widely used cross-entropy loss. Empirical results from the experiments showed that on average, the Supervised Contrastive Learning model performed 6.05% better than the baseline model on the 7 downstream datasets. The findings suggest that Supervised Contrastive Learning models can potentially learn more robust representations that transfer better across domains than cross-entropy models when trained on a multi-domain dataset.