Label Anchored Contrastive Learning for Language Understanding

TL;DR

This paper introduces LaCon, a label anchored contrastive learning method for language understanding that improves classification performance by leveraging class labels for hard positive/negative mining without extra data or architecture.

Contribution

The paper proposes a novel contrastive learning approach (LaCon) that incorporates label information through multiple contrastive objectives, enhancing language understanding tasks.

Findings

Up to 4.1% accuracy improvement on GLUE and CLUE datasets.

Significant gains in few-shot and data imbalance scenarios, up to 9.4%.

Compatible with existing pre-trained language models without extra architecture.

Abstract

Contrastive learning (CL) has achieved astonishing progress in computer vision, speech, and natural language processing fields recently with self-supervised learning. However, CL approach to the supervised setting is not fully explored, especially for the natural language understanding classification task. Intuitively, the class label itself has the intrinsic ability to perform hard positive/negative mining, which is crucial for CL. Motivated by this, we propose a novel label anchored contrastive learning approach (denoted as LaCon) for language understanding. Specifically, three contrastive objectives are devised, including a multi-head instance-centered contrastive loss (ICL), a label-centered contrastive loss (LCL), and a label embedding regularizer (LER). Our approach does not require any specialized network architecture or any extra data augmentation, thus it can be easily plugged into existing powerful pre-trained language models. Compared to the state-of-the-art baselines, LaCon obtains up to 4.1% improvement on the popular datasets of GLUE and CLUE benchmarks. Besides, LaCon also demonstrates significant advantages under the few-shot and data imbalance settings, which obtains up to 9.4% improvement on the FewGLUE and FewCLUE benchmarking tasks.