Semi-supervised Multimodal Representation Learning through a Global Workspace

TL;DR

This paper introduces a global workspace architecture for multimodal learning that aligns and translates between modalities with minimal supervised data, inspired by cognitive theories, and demonstrates its effectiveness across vision-language tasks.

Contribution

The paper proposes a novel global workspace model for multimodal learning that uses self-supervised cycle-consistency, reducing the need for large labeled datasets and improving transfer capabilities.

Findings

Achieves multimodal alignment with 4-7 times less supervised data.

Effective for downstream classification and transfer learning.

Both shared workspace and cycle-consistency are crucial for performance.

Abstract

Recent deep learning models can efficiently combine inputs from different modalities (e.g., images and text) and learn to align their latent representations, or to translate signals from one domain to another (as in image captioning, or text-to-image generation). However, current approaches mainly rely on brute-force supervised training over large multimodal datasets. In contrast, humans (and other animals) can learn useful multimodal representations from only sparse experience with matched cross-modal data. Here we evaluate the capabilities of a neural network architecture inspired by the cognitive notion of a "Global Workspace": a shared representation for two (or more) input modalities. Each modality is processed by a specialized system (pretrained on unimodal data, and subsequently frozen). The corresponding latent representations are then encoded to and decoded from a single shared workspace. Importantly, this architecture is amenable to self-supervised training via cycle-consistency: encoding-decoding sequences should approximate the identity function. For various pairings of vision-language modalities and across two datasets of varying complexity, we show that such an architecture can be trained to align and translate between two modalities with very little need for matched data (from 4 to 7 times less than a fully supervised approach). The global workspace representation can be used advantageously for downstream classification tasks and for robust transfer learning. Ablation studies reveal that both the shared workspace and the self-supervised cycle-consistency training are critical to the system's performance.