Simultaneous Long-tailed Recognition and Multi-modal Fusion for Highly Imbalanced Multi-modal Data

TL;DR

This paper presents a novel multi-modal framework for long-tailed recognition that dynamically fuses heterogeneous data sources, improving classification performance on imbalanced datasets.

Contribution

It introduces a modality-aware multi-expert architecture with confidence-guided fusion and specialized training procedures for multi-modal long-tailed recognition.

Findings

Outperforms existing methods on benchmark datasets.

Effectively integrates multi-modal data for imbalanced classification.

Demonstrates robustness and generalization in real-world scenarios.

Abstract

Long-tailed distributions in class-imbalanced data present a fundamental challenge for deep learning models, which tend to be biased toward majority classes. While recent methods for long-tailed recognition have mitigated this issue, they are largely restricted to single-modal inputs and cannot fully exploit complementary information from diverse data sources. In this work, we introduce a new framework for long-tailed recognition that explicitly handles multi-modal inputs. Our approach extends multi-expert architectures to the multi-modal setting by fusing heterogeneous data into a unified representation while leveraging modality-specific networks to estimate the informativeness of each modality. These confidence-guided weights dynamically modulate the fusion process, ensuring that more informative modalities contribute more strongly to the final decision. To further enhance performance, we design specialized training and test procedures that accommodate diverse modality combinations, including images and tabular data. Extensive experiments on benchmark and real-world datasets demonstrate that the proposed approach not only effectively integrates multi-modal information but also outperforms existing methods in handling long-tailed, class-imbalanced scenarios, highlighting its robustness and generalization capability.