Mixture-of-Modality-Experts with Holistic Token Learning for Fine-Grained Multimodal Visual Analytics in Driver Action Recognition

TL;DR

This paper introduces a novel adaptive multimodal learning framework called MoME with HTL, enhancing fine-grained driver action recognition by improving expert collaboration and interpretability.

Contribution

The paper proposes a flexible MoME framework with HTL that dynamically adapts modality collaboration and enhances intra- and inter-expert learning for multimodal tasks.

Findings

Outperforms single-modal and existing multimodal baselines on driver action recognition.

HTL improves subtle multimodal understanding and interpretability.

Ablation studies confirm the effectiveness of the proposed strategies.

Abstract

Robust multimodal visual analytics remains challenging when heterogeneous modalities provide complementary but input-dependent evidence for decision-making.Existing multimodal learning methods mainly rely on fixed fusion modules or predefined cross-modal interactions, which are often insufficient to adapt to changing modality reliability and to capture fine-grained action cues. To address this issue, we propose a Mixture-of-Modality-Experts (MoME) framework with a Holistic Token Learning (HTL) strategy. MoME enables adaptive collaboration among modality-specific experts, while HTL improves both intra-expert refinement and inter-expert knowledge transfer through class tokens and spatio-temporal tokens. In this way, our method forms a knowledge-centric multimodal learning framework that improves expert specialization while reducing ambiguity in multimodal fusion.We validate the proposed framework on driver action recognition as a representative multimodal understanding taskThe experimental results on the public benchmark show that the proposed MoME framework and the HTL strategy jointly outperform representative single-modal and multimodal baselines. Additional ablation, validation, and visualization results further verify that the proposed HTL strategy improves subtle multimodal understanding and offers better interpretability.