i-Mix: A Domain-Agnostic Strategy for Contrastive Representation Learning

TL;DR

i-Mix introduces a domain-agnostic regularization technique for contrastive learning by mixing data and labels, enhancing representation quality across diverse data types like images, speech, and tabular data.

Contribution

The paper proposes i-Mix, a novel domain-agnostic regularization method that improves contrastive learning by mixing data and virtual labels, applicable across various data domains.

Findings

i-Mix improves representation quality across image, speech, and tabular data.

It acts as an effective regularizer confirmed by ablation studies.

The method is simple, effective, and domain-agnostic.

Abstract

Contrastive representation learning has shown to be effective to learn representations from unlabeled data. However, much progress has been made in vision domains relying on data augmentations carefully designed using domain knowledge. In this work, we propose i-Mix, a simple yet effective domain-agnostic regularization strategy for improving contrastive representation learning. We cast contrastive learning as training a non-parametric classifier by assigning a unique virtual class to each data in a batch. Then, data instances are mixed in both the input and virtual label spaces, providing more augmented data during training. In experiments, we demonstrate that i-Mix consistently improves the quality of learned representations across domains, including image, speech, and tabular data. Furthermore, we confirm its regularization effect via extensive ablation studies across model and dataset sizes. The code is available at https://github.com/kibok90/imix.