Identical and Fraternal Twins: Fine-Grained Semantic Contrastive Learning of Sentence Representations

TL;DR

This paper introduces IFTCL, a novel contrastive learning framework for sentence representations that preserves semantic margins and improves performance by addressing limitations of existing methods.

Contribution

The paper proposes the Twins Loss and hippocampus queue mechanism, enhancing contrastive learning by maintaining semantic margins and reusing negative samples efficiently.

Findings

Outperforms state-of-the-art on nine semantic textual similarity tasks

Effectively preserves semantic margins during training

Improves efficiency with hippocampus queue mechanism

Abstract

The enhancement of unsupervised learning of sentence representations has been significantly achieved by the utility of contrastive learning. This approach clusters the augmented positive instance with the anchor instance to create a desired embedding space. However, relying solely on the contrastive objective can result in sub-optimal outcomes due to its inability to differentiate subtle semantic variations between positive pairs. Specifically, common data augmentation techniques frequently introduce semantic distortion, leading to a semantic margin between the positive pair. While the InfoNCE loss function overlooks the semantic margin and prioritizes similarity maximization between positive pairs during training, leading to the insensitive semantic comprehension ability of the trained model. In this paper, we introduce a novel Identical and Fraternal Twins of Contrastive Learning (named IFTCL) framework, capable of simultaneously adapting to various positive pairs generated by different augmentation techniques. We propose a \textit{Twins Loss} to preserve the innate margin during training and promote the potential of data enhancement in order to overcome the sub-optimal issue. We also present proof-of-concept experiments combined with the contrastive objective to prove the validity of the proposed Twins Loss. Furthermore, we propose a hippocampus queue mechanism to restore and reuse the negative instances without additional calculation, which further enhances the efficiency and performance of the IFCL. We verify the IFCL framework on nine semantic textual similarity tasks with both English and Chinese datasets, and the experimental results show that IFCL outperforms state-of-the-art methods.