Cross-Modal Discrete Representation Learning

TL;DR

This paper introduces a self-supervised framework for learning fine-grained, cross-modal representations using a shared discretized embedding space, enabling improved cross-modal retrieval and localization without supervision.

Contribution

It proposes a novel discretized embedding space and a cross-modal code matching objective for fine-grained, multi-modal representation learning.

Findings

Discretized representations improve cross-modal retrieval performance.

Shared embedding space captures semantic concepts across modalities.

Discretized clusters align with semantic concepts across different data types.

Abstract

Recent advances in representation learning have demonstrated an ability to represent information from different modalities such as video, text, and audio in a single high-level embedding vector. In this work we present a self-supervised learning framework that is able to learn a representation that captures finer levels of granularity across different modalities such as concepts or events represented by visual objects or spoken words. Our framework relies on a discretized embedding space created via vector quantization that is shared across different modalities. Beyond the shared embedding space, we propose a Cross-Modal Code Matching objective that forces the representations from different views (modalities) to have a similar distribution over the discrete embedding space such that cross-modal objects/actions localization can be performed without direct supervision. In our experiments we show that the proposed discretized multi-modal fine-grained representation (e.g., pixel/word/frame) can complement high-level summary representations (e.g., video/sentence/waveform) for improved performance on cross-modal retrieval tasks. We also observe that the discretized representation uses individual clusters to represent the same semantic concept across modalities.