Can Contrastive Learning Refine Embeddings

TL;DR

This paper introduces SIMSKIP, a contrastive learning framework that refines input embeddings using encoder outputs, improving downstream task performance without increasing error bounds.

Contribution

The paper presents a novel contrastive learning method that leverages encoder output embeddings to enhance input representations for downstream tasks.

Findings

SIMSKIP improves downstream task performance.

Theoretical analysis shows no increased error bounds.

Experimental results validate effectiveness across datasets.

Abstract

Recent advancements in contrastive learning have revolutionized self-supervised representation learning and achieved state-of-the-art performance on benchmark tasks. While most existing methods focus on applying contrastive learning to input data modalities such as images, natural language sentences, or networks, they overlook the potential of utilizing outputs from previously trained encoders. In this paper, we introduce SIMSKIP, a novel contrastive learning framework that specifically refines input embeddings for downstream tasks. Unlike traditional unsupervised learning approaches, SIMSKIP takes advantage of the output embeddings of encoder models as its input. Through theoretical analysis, we provide evidence that applying SIMSKIP does not result in larger upper bounds on downstream task errors than those of the original embeddings, which serve as SIMSKIP's input. Experimental results on various open datasets demonstrate that the embeddings produced by SIMSKIP improve performance on downstream tasks.