Dissecting Representation Misalignment in Contrastive Learning via Influence Function

TL;DR

This paper introduces ECIF, an influence function tailored for contrastive learning, enabling efficient detection of data misalignments and improving interpretability of large-scale multimodal models like CLIP.

Contribution

The paper presents ECIF, a novel influence function for contrastive loss that considers both positive and negative samples, enhancing data valuation and model transparency.

Findings

ECIF provides accurate influence estimates without retraining.

ECIF improves detection of data misalignments in contrastive models.

Experimental results show ECIF outperforms baseline methods in interpretability.

Abstract

Contrastive learning, commonly applied in large-scale multimodal models, often relies on data from diverse and often unreliable sources, which can include misaligned or mislabeled text-image pairs. This frequently leads to robustness issues and hallucinations, ultimately causing performance degradation. Data valuation is an efficient way to detect and trace these misalignments. Nevertheless, existing methods are computationally expensive for large-scale models. Although computationally efficient, classical influence functions are inadequate for contrastive learning models, as they were initially designed for pointwise loss. Furthermore, contrastive learning involves minimizing the distance between positive sample modalities while maximizing the distance between negative sample modalities. This necessitates evaluating the influence of samples from both perspectives. To tackle these challenges, we introduce the Extended Influence Function for Contrastive Loss (ECIF), an influence function crafted for contrastive loss. ECIF considers both positive and negative samples and provides a closed-form approximation of contrastive learning models, eliminating the need for retraining. Building upon ECIF, we develop a series of algorithms for data evaluation, misalignment detection, and misprediction trace-back tasks. Experimental results demonstrate our ECIF advances the transparency and interpretability of CLIP-style embedding models by offering a more accurate assessment of data impact and model alignment compared to traditional baseline methods.