Joint Wasserstein Autoencoders for Aligning Multimodal Embeddings

TL;DR

This paper introduces a joint Wasserstein autoencoder framework that aligns multimodal embeddings, such as images and text, by enforcing shared Gaussian priors in the latent space, leading to improved cross-modal retrieval and localization.

Contribution

It proposes a novel semi-supervised joint Gaussian regularization method within Wasserstein autoencoders for better cross-modal semantic alignment.

Findings

Achieves state-of-the-art accuracy in cross-modal retrieval and phrase localization.

Demonstrates significantly improved generalization across datasets.

Ensures semantic continuity in the latent space for multimodal data.

Abstract

One of the key challenges in learning joint embeddings of multiple modalities, e.g. of images and text, is to ensure coherent cross-modal semantics that generalize across datasets. We propose to address this through joint Gaussian regularization of the latent representations. Building on Wasserstein autoencoders (WAEs) to encode the input in each domain, we enforce the latent embeddings to be similar to a Gaussian prior that is shared across the two domains, ensuring compatible continuity of the encoded semantic representations of images and texts. Semantic alignment is achieved through supervision from matching image-text pairs. To show the benefits of our semi-supervised representation, we apply it to cross-modal retrieval and phrase localization. We not only achieve state-of-the-art accuracy, but significantly better generalization across datasets, owing to the semantic continuity of the latent space.