Many could be better than all: A novel instance-oriented algorithm for Multi-modal Multi-label problem

TL;DR

This paper introduces an instance-oriented multi-modal classifier chain algorithm for multi-modal multi-label problems, demonstrating that selectively using relevant modalities per instance can outperform using all modalities.

Contribution

The novel MCC algorithm adaptively extracts relevant modalities for each instance, improving prediction accuracy in multi-modal multi-label tasks.

Findings

MCC outperforms baseline methods on real-world and public datasets.

Selective modality extraction can be more effective than using all available modalities.

The approach is robust across different types of multi-modal data.

Abstract

With the emergence of diverse data collection techniques, objects in real applications can be represented as multi-modal features. What's more, objects may have multiple semantic meanings. Multi-modal and Multi-label (MMML) problem becomes a universal phenomenon. The quality of data collected from different channels are inconsistent and some of them may not benefit for prediction. In real life, not all the modalities are needed for prediction. As a result, we propose a novel instance-oriented Multi-modal Classifier Chains (MCC) algorithm for MMML problem, which can make convince prediction with partial modalities. MCC extracts different modalities for different instances in the testing phase. Extensive experiments are performed on one real-world herbs dataset and two public datasets to validate our proposed algorithm, which reveals that it may be better to extract many instead of all of the modalities at hand.