MODA: MOdular Duplex Attention for Multimodal Perception, Cognition, and Emotion Understanding

TL;DR

This paper introduces MODA, a novel attention mechanism for multimodal models that improves high-level perception, cognition, and emotion understanding by addressing attention deficits through modular duplex attention and alignment strategies.

Contribution

The paper proposes MODA, a new attention mechanism with duplex attention and alignment strategies, enhancing multimodal learning for complex perception and cognition tasks.

Findings

MODA outperforms existing methods on 21 benchmark datasets.

MODA effectively improves fine-grained cognition and emotion understanding.

The approach enhances cross-modal interaction and attention consistency.

Abstract

Multimodal large language models (MLLMs) recently showed strong capacity in integrating data among multiple modalities, empowered by a generalizable attention architecture. Advanced methods predominantly focus on language-centric tuning while less exploring multimodal tokens mixed through attention, posing challenges in high-level tasks that require fine-grained cognition and emotion understanding. In this work, we identify the attention deficit disorder problem in multimodal learning, caused by inconsistent cross-modal attention and layer-by-layer decayed attention activation. To address this, we propose a novel attention mechanism, termed MOdular Duplex Attention (MODA), simultaneously conducting the inner-modal refinement and inter-modal interaction. MODA employs a correct-after-align strategy to effectively decouple modality alignment from cross-layer token mixing. In the alignment phase, tokens are mapped to duplex modality spaces based on the basis vectors, enabling the interaction between visual and language modality. Further, the correctness of attention scores is ensured through adaptive masked attention, which enhances the model's flexibility by allowing customizable masking patterns for different modalities. Extensive experiments on 21 benchmark datasets verify the effectiveness of MODA in perception, cognition, and emotion tasks. Source code and demo are available in https://zzcheng.top/MODA.