DRDF: Determining the Importance of Different Multimodal Information with Dual-Router Dynamic Framework

TL;DR

This paper introduces DRDF, a flexible framework that dynamically determines the importance of text and image information in multimodal tasks, improving performance across various datasets and backbones.

Contribution

The paper proposes a novel Dual-Router Dynamic Framework (DRDF) that adaptively weights modal information for enhanced multimodal task performance.

Findings

DRDF outperforms all baseline methods on multiple datasets.

The framework is highly general and compatible with various backbones.

Component analysis confirms the effectiveness of the dynamic weighting mechanism.

Abstract

In multimodal tasks, we find that the importance of text and image modal information is different for different input cases, and for this motivation, we propose a high-performance and highly general Dual-Router Dynamic Framework (DRDF), consisting of Dual-Router, MWF-Layer, experts and expert fusion unit. The text router and image router in Dual-Router accept text modal information and image modal information, and use MWF-Layer to determine the importance of modal information. Based on the result of the determination, MWF-Layer generates fused weights for the fusion of experts. Experts are model backbones that match the current task. DRDF has high performance and high generality, and we have tested 12 backbones such as Visual BERT on multimodal dataset Hateful memes, unimodal dataset CIFAR10, CIFAR100, and TinyImagenet. Our DRDF outperforms all the baselines. We also verified the components of DRDF in detail by ablations, compared and discussed the reasons and ideas of DRDF design.