Embedding Geometries of Contrastive Language-Image Pre-Training

TL;DR

This paper explores alternative geometric frameworks for contrastive language-image pre-training, demonstrating that Euclidean geometry variants can outperform or match CLIP's performance and better support hierarchical relationships.

Contribution

It introduces EuCLIP, a Euclidean geometry-based variant of CLIP, showing its effectiveness and advantages over traditional cosine similarity approaches.

Findings

EuCLIP matches or exceeds CLIP performance

EuCLIP supports hierarchical relationships effectively

Alternative geometries can improve contrastive pre-training

Abstract

Since the publication of CLIP, the approach of using InfoNCE loss for contrastive pre-training has become widely popular for bridging two or more modalities. Despite its wide adoption, CLIP's original design choices of L2 normalization and cosine similarity logit have rarely been revisited. We have systematically experimented with alternative geometries and softmax logits for language-image pre-training and identified that variants with intuitive Euclidean geometry, Euclidean CLIP (EuCLIP), match or exceed the performance of CLIP and support hierarchical relationships at least as well as more complicated hyperbolic alternative.