Switch-BERT: Learning to Model Multimodal Interactions by Switching Attention and Input

TL;DR

Switch-BERT introduces a flexible, learnable multimodal interaction model that dynamically switches attention modes and layers, improving performance across vision-language tasks by mitigating modality mismatch.

Contribution

It extends BERT with learnable layer-wise and cross-layer interactions, enabling adaptive modeling of intra- and inter-modal relationships in multimodal learning.

Findings

Outperforms or matches state-of-the-art in VQA, image-text retrieval, and referring expression tasks.

Learns to attend outputs from various depths, reducing modality mismatch.

Achieves superior task-specific multimodal interaction modeling.

Abstract

The ability to model intra-modal and inter-modal interactions is fundamental in multimodal machine learning. The current state-of-the-art models usually adopt deep learning models with fixed structures. They can achieve exceptional performances on specific tasks, but face a particularly challenging problem of modality mismatch because of diversity of input modalities and their fixed structures. In this paper, we present \textbf{Switch-BERT} for joint vision and language representation learning to address this problem. Switch-BERT extends BERT architecture by introducing learnable layer-wise and cross-layer interactions. It learns to optimize attention from a set of attention modes representing these interactions. One specific property of the model is that it learns to attend outputs from various depths, therefore mitigates the modality mismatch problem. We present extensive experiments on visual question answering, image-text retrieval and referring expression comprehension experiments. Results confirm that, whereas alternative architectures including ViLBERT and UNITER may excel in particular tasks, Switch-BERT can consistently achieve better or comparable performances than the current state-of-the-art models in these tasks. Ablation studies indicate that the proposed model achieves superior performances due to its ability in learning task-specific multimodal interactions.