Virtual embeddings and self-consistency for self-supervised learning

TL;DR

This paper introduces TriMix, a novel self-supervised learning method that generates virtual embeddings via linear interpolation and enforces self-consistency, leading to improved performance across diverse datasets and tasks.

Contribution

TriMix is the first approach to utilize latent space augmentation through virtual embeddings and self-consistency in SSL, enhancing representation learning and transferability.

Findings

Achieved 2.71% and 0.41% improvements over SOTA on benchmark datasets.

Outperformed existing methods in semi-supervised learning, especially with limited data.

Pre-trained models demonstrated superior transfer learning capabilities.

Abstract

Self-supervised Learning (SSL) has recently gained much attention due to the high cost and data limitation in the training of supervised learning models. The current paradigm in the SSL is to utilize data augmentation at the input space to create different views of the same images and train a model to maximize the representations between similar images and minimize them for different ones. While this approach achieves state-of-the-art (SOTA) results in various downstream tasks, it still lakes the opportunity to investigate the latent space augmentation. This paper proposes TriMix, a novel concept for SSL that generates virtual embeddings through linear interpolation of the data, thus providing the model with novel representations. Our strategy focuses on training the model to extract the original embeddings from virtual ones, hence, better representation learning. Additionally, we propose a self-consistency term that improves the consistency between the virtual and actual embeddings. We validate TriMix on eight benchmark datasets consisting of natural and medical images with an improvement of 2.71% and 0.41% better than the second-best models for both data types. Further, our approach outperformed the current methods in semi-supervised learning, particularly in low data regimes. Besides, our pre-trained models showed better transfer to other datasets.