Multimodal Token Fusion for Vision Transformers

TL;DR

This paper introduces TokenFusion, a novel multimodal token fusion method for vision transformers that enhances multi-modal learning by dynamically replacing uninformative tokens with inter-modal features, improving performance across various vision tasks.

Contribution

TokenFusion is a new method that effectively fuses multiple modalities in vision transformers by dynamically replacing tokens, maintaining architecture simplicity while improving multimodal task performance.

Findings

TokenFusion outperforms state-of-the-art methods in image translation, semantic segmentation, and 3D object detection.

The method effectively detects and replaces uninformative tokens with inter-modal features.

Experiments demonstrate significant performance gains across diverse multimodal vision tasks.

Abstract

Many adaptations of transformers have emerged to address the single-modal vision tasks, where self-attention modules are stacked to handle input sources like images. Intuitively, feeding multiple modalities of data to vision transformers could improve the performance, yet the inner-modal attentive weights may also be diluted, which could thus undermine the final performance. In this paper, we propose a multimodal token fusion method (TokenFusion), tailored for transformer-based vision tasks. To effectively fuse multiple modalities, TokenFusion dynamically detects uninformative tokens and substitutes these tokens with projected and aggregated inter-modal features. Residual positional alignment is also adopted to enable explicit utilization of the inter-modal alignments after fusion. The design of TokenFusion allows the transformer to learn correlations among multimodal features, while the single-modal transformer architecture remains largely intact. Extensive experiments are conducted on a variety of homogeneous and heterogeneous modalities and demonstrate that TokenFusion surpasses state-of-the-art methods in three typical vision tasks: multimodal image-to-image translation, RGB-depth semantic segmentation, and 3D object detection with point cloud and images. Our code is available at https://github.com/yikaiw/TokenFusion.