Jina CLIP: Your CLIP Model Is Also Your Text Retriever

TL;DR

Jina CLIP introduces a multi-task contrastive training approach that enables a single CLIP-based model to excel in both multimodal and text-only retrieval tasks, streamlining information retrieval systems.

Contribution

The paper presents a novel multi-task contrastive training method that improves CLIP's performance on text-only tasks without sacrificing multimodal retrieval capabilities.

Findings

Achieved state-of-the-art results on text-image retrieval

Improved text-only retrieval performance

Unified model for multimodal and text-only tasks

Abstract

Contrastive Language-Image Pretraining (CLIP) is widely used to train models to align images and texts in a common embedding space by mapping them to fixed-sized vectors. These models are key to multimodal information retrieval and related tasks. However, CLIP models generally underperform in text-only tasks compared to specialized text models. This creates inefficiencies for information retrieval systems that keep separate embeddings and models for text-only and multimodal tasks. We propose a novel, multi-task contrastive training method to address this issue, which we use to train the jina-clip-v1 model to achieve the state-of-the-art performance on both text-image and text-text retrieval tasks.