Attention Bottlenecks for Multimodal Fusion

TL;DR

This paper proposes a transformer architecture with fusion bottlenecks for multimodal fusion, improving performance and efficiency in audio-visual classification tasks by condensing relevant information at multiple layers.

Contribution

Introduces a novel transformer model with fusion bottlenecks for multi-layer modality fusion, enhancing accuracy and reducing computational costs.

Findings

Achieves state-of-the-art results on multiple benchmarks.

Reduces computational cost compared to traditional methods.

Improves multimodal fusion performance.

Abstract

Humans perceive the world by concurrently processing and fusing high-dimensional inputs from multiple modalities such as vision and audio. Machine perception models, in stark contrast, are typically modality-specific and optimised for unimodal benchmarks, and hence late-stage fusion of final representations or predictions from each modality (`late-fusion') is still a dominant paradigm for multimodal video classification. Instead, we introduce a novel transformer based architecture that uses `fusion bottlenecks' for modality fusion at multiple layers. Compared to traditional pairwise self-attention, our model forces information between different modalities to pass through a small number of bottleneck latents, requiring the model to collate and condense the most relevant information in each modality and only share what is necessary. We find that such a strategy improves fusion performance, at the same time reducing computational cost. We conduct thorough ablation studies, and achieve state-of-the-art results on multiple audio-visual classification benchmarks including Audioset, Epic-Kitchens and VGGSound. All code and models will be released.