Virtual Augmentation Supported Contrastive Learning of Sentence Representations

TL;DR

This paper introduces VaSCL, a novel contrastive learning method for sentence representations that uses virtual augmentations generated from in-batch neighbors, achieving state-of-the-art results in unsupervised settings.

Contribution

The paper proposes a new virtual augmentation technique for contrastive learning in NLP that leverages in-batch neighbors to improve sentence representations.

Findings

VaSCL achieves new state-of-the-art performance on multiple downstream tasks.

The virtual augmentation method enhances contrastive learning without domain-specific data augmentation.

Using in-batch neighbors effectively approximates neighborhoods for augmentation.

Abstract

Despite profound successes, contrastive representation learning relies on carefully designed data augmentations using domain specific knowledge. This challenge is magnified in natural language processing where no general rules exist for data augmentation due to the discrete nature of natural language. We tackle this challenge by presenting a Virtual augmentation Supported Contrastive Learning of sentence representations (VaSCL). Originating from the interpretation that data augmentation essentially constructs the neighborhoods of each training instance, we in turn utilize the neighborhood to generate effective data augmentations. Leveraging the large training batch size of contrastive learning, we approximate the neighborhood of an instance via its K-nearest in-batch neighbors in the representation space. We then define an instance discrimination task regarding this neighborhood and generate the virtual augmentation in an adversarial training manner. We access the performance of VaSCL on a wide range of downstream tasks, and set a new state-of-the-art for unsupervised sentence representation learning.