SparseCL: Sparse Contrastive Learning for Contradiction Retrieval

TL;DR

SparseCL introduces a novel sparse contrastive learning method that enhances contradiction retrieval by efficiently capturing subtle contradictory nuances, significantly improving accuracy and speed over existing methods in large-scale document retrieval tasks.

Contribution

The paper proposes SparseCL, a new contrastive learning approach that uses sparse sentence embeddings to better identify contradictions, addressing limitations of similarity search and cross-encoder models.

Findings

Over 30% accuracy improvement on MSMARCO and HotpotQA datasets.

Enhanced contradiction detection speed through reduced vector comparisons.

Effective in cleaning corrupted corpora for high-quality QA retrieval.

Abstract

Contradiction retrieval refers to identifying and extracting documents that explicitly disagree with or refute the content of a query, which is important to many downstream applications like fact checking and data cleaning. To retrieve contradiction argument to the query from large document corpora, existing methods such as similarity search and crossencoder models exhibit significant limitations. The former struggles to capture the essence of contradiction due to its inherent nature of favoring similarity, while the latter suffers from computational inefficiency, especially when the size of corpora is large. To address these challenges, we introduce a novel approach: SparseCL that leverages specially trained sentence embeddings designed to preserve subtle, contradictory nuances between sentences. Our method utilizes a combined metric of cosine similarity and a sparsity function to efficiently identify and retrieve documents that contradict a given query. This approach dramatically enhances the speed of contradiction detection by reducing the need for exhaustive document comparisons to simple vector calculations. We validate our model using the Arguana dataset, a benchmark dataset specifically geared towards contradiction retrieval, as well as synthetic contradictions generated from the MSMARCO and HotpotQA datasets using GPT-4. Our experiments demonstrate the efficacy of our approach not only in contradiction retrieval with more than 30% accuracy improvements on MSMARCO and HotpotQA across different model architectures but also in applications such as cleaning corrupted corpora to restore high-quality QA retrieval. This paper outlines a promising direction for improving the accuracy and efficiency of contradiction retrieval in large-scale text corpora.