See, Hear, and Read: Deep Aligned Representations

TL;DR

This paper introduces a deep learning approach that learns shared representations across vision, sound, and language modalities using large-scale synchronized data, enabling effective cross-modal retrieval and transfer learning.

Contribution

It presents a novel joint training method for aligned multimodal representations that transfer knowledge across modalities without direct training data for some pairs.

Findings

Representation improves cross-modal retrieval tasks.

Model enables transfer between text and sound modalities.

Hidden units automatically detect concepts across modalities.

Abstract

We capitalize on large amounts of readily-available, synchronous data to learn a deep discriminative representations shared across three major natural modalities: vision, sound and language. By leveraging over a year of sound from video and millions of sentences paired with images, we jointly train a deep convolutional network for aligned representation learning. Our experiments suggest that this representation is useful for several tasks, such as cross-modal retrieval or transferring classifiers between modalities. Moreover, although our network is only trained with image+text and image+sound pairs, it can transfer between text and sound as well, a transfer the network never observed during training. Visualizations of our representation reveal many hidden units which automatically emerge to detect concepts, independent of the modality.