Gated Recursive Fusion: A Stateful Approach to Scalable Multimodal Transformers

TL;DR

Gated Recursive Fusion (GRF) introduces a scalable, recurrent multimodal transformer architecture that processes multiple modalities sequentially, maintaining competitive performance while significantly reducing computational complexity.

Contribution

The paper proposes GRF, a novel linear-scaling, stateful fusion method that combines cross-modal attention with a gated recurrent mechanism for efficient multimodal learning.

Findings

Achieves competitive results on CMU-MOSI benchmark.

Creates structured, class-separable representations.

Scales linearly with the number of modalities.

Abstract

Multimodal learning faces a fundamental tension between deep, fine-grained fusion and computational scalability. While cross-attention models achieve strong performance through exhaustive pairwise fusion, their quadratic complexity is prohibitive for settings with many modalities. We address this challenge with Gated Recurrent Fusion (GRF), a novel architecture that captures the power of cross-modal attention within a linearly scalable, recurrent pipeline. Our method processes modalities sequentially, updating an evolving multimodal context vector at each step. The core of our approach is a fusion block built on Transformer Decoder layers that performs symmetric cross-attention, mutually enriching the shared context and the incoming modality. This enriched information is then integrated via a Gated Fusion Unit (GFU) a GRU-inspired mechanism that dynamically arbitrates information flow, enabling the model to selectively retain or discard features. This stateful, recurrent design scales linearly with the number of modalities, O(n), making it ideal for high-modality environments. Experiments on the CMU-MOSI benchmark demonstrate that GRF achieves competitive performance compared to more complex baselines. Visualizations of the embedding space further illustrate that GRF creates structured, class-separable representations through its progressive fusion mechanism. Our work presents a robust and efficient paradigm for powerful, scalable multimodal representation learning.