Enhancing CLIP Conceptual Embedding through Knowledge Distillation

TL;DR

This paper introduces Knowledge-CLIP, a novel method that enhances CLIP's multi-modal embedding capabilities by applying knowledge distillation from Llama 2, improving both text and image representations through specialized training objectives.

Contribution

It proposes a new knowledge distillation framework for CLIP that incorporates Llama 2, including text embedding distillation, concept learning via clustering, and contrastive learning.

Findings

Improved performance of text and image encoders.

Effective integration of Llama 2 knowledge into CLIP.

Enhanced multi-modal embedding quality.

Abstract

Recently, CLIP has become an important model for aligning images and text in multi-modal contexts. However, researchers have identified limitations in the ability of CLIP's text and image encoders to extract detailed knowledge from pairs of captions and images. In response, this paper presents Knowledge-CLIP, an innovative approach designed to improve CLIP's performance by integrating a new knowledge distillation (KD) method based on Llama 2. Our approach focuses on three key objectives: Text Embedding Distillation, Concept Learning, and Contrastive Learning. First, Text Embedding Distillation involves training the Knowledge-CLIP text encoder to mirror the teacher model, Llama 2. Next, Concept Learning assigns a soft concept label to each caption-image pair by employing offline K-means clustering on text data from Llama 2, enabling Knowledge-CLIP to learn from these soft concept labels. Lastly, Contrastive Learning aligns the text and image embeddings. Our experimental findings show that the proposed model improves the performance of both text and image encoders.