Contrastive vision-language learning with paraphrasing and negation

TL;DR

This paper introduces SemCLIP, a contrastive learning method that improves vision-language models' robustness to paraphrasing and negation by using a new loss function and LLM-generated training triples, enhancing image retrieval and classification accuracy.

Contribution

SemCLIP's novel loss function and training approach effectively distinguish negated captions from original images while maintaining performance on paraphrased captions, improving robustness to semantic variations.

Findings

SemCLIP improves negation robustness, increasing accuracy from 68.1% to 78.1% on CC-Neg.

SemCLIP maintains comparable performance to CLIP on original caption retrieval.

SemCLIP outperforms models trained with negated captions on downstream tasks.

Abstract

Contrastive vision-language models continue to be the dominant approach for image and text retrieval. Contrastive Language-Image Pre-training (CLIP) trains two neural networks in contrastive manner to align their image and text embeddings in a shared latent space. Recent results evaluating CLIP on negated or paraphrased text have shown mixed performance because negation changes meaning radically with minimal lexical changes, while paraphrasing can create very different textual expressions with the same intended meaning. This poses a significant challenge for improving the evaluation results and alignment of vision-language models. To address this challenge, this paper evaluates the combination of paraphrasing and negation, proposes a new CLIP contrastive loss function accounting for both paraphrasing and negation, and applies LLM-generated training triples consisting of original, paraphrased and negated textual captions to CLIP-like training models. The approach, called SemCLIP, is shown to move paraphrased captions towards the original image embeddings while pushing negated captions further away in embedding space. Empirically, SemCLIP is shown to be capable of preserving CLIP's performance while increasing considerably the distances to negated captions. On the CC-Neg benchmark using an original over negation image-retrieval accuracy metric, SemCLIP improves accuracy from 68.1% to 78.1%. Although results are mixed when compared with CLIP on the Sugarcrepe++ benchmark, SemCLIP's performance is generally better than the models trained with negated captions. This robustness to negation extends to downstream zero-shot classification tasks where SemCLIP pre-trained on Sugarcrepe++ performs better than CLIP on all tested downstream tasks. These results indicate that SemCLIP can achieve significant robustness to semantic transformations.