CrossCLR: Cross-modal Contrastive Learning For Multi-modal Video Representations

TL;DR

CrossCLR introduces a novel contrastive loss for multi-modal video representations that considers intra-modality similarities and avoids false negatives, significantly improving video-text retrieval and captioning performance.

Contribution

It proposes a new contrastive loss that accounts for intra-modality similarities and excludes related samples, enhancing multi-modal embedding quality.

Findings

Improved video-text retrieval on Youcook2 and LSMDC datasets.

Enhanced video captioning performance on Youcook2.

General applicability to other modality pairs.

Abstract

Contrastive learning allows us to flexibly define powerful losses by contrasting positive pairs from sets of negative samples. Recently, the principle has also been used to learn cross-modal embeddings for video and text, yet without exploiting its full potential. In particular, previous losses do not take the intra-modality similarities into account, which leads to inefficient embeddings, as the same content is mapped to multiple points in the embedding space. With CrossCLR, we present a contrastive loss that fixes this issue. Moreover, we define sets of highly related samples in terms of their input embeddings and exclude them from the negative samples to avoid issues with false negatives. We show that these principles consistently improve the quality of the learned embeddings. The joint embeddings learned with CrossCLR extend the state of the art in video-text retrieval on Youcook2 and LSMDC datasets and in video captioning on Youcook2 dataset by a large margin. We also demonstrate the generality of the concept by learning improved joint embeddings for other pairs of modalities.