Cross-Modality Clustering-based Self-Labeling for Multimodal Data Classification

TL;DR

This paper introduces CMCSL, a semi-supervised method that leverages cross-modality clustering to improve labeling and classification accuracy in multimodal data, especially with limited labeled data.

Contribution

The paper presents a novel cross-modality clustering-based self-labeling approach that propagates labels across modalities to enhance semi-supervised learning in multimodal classification.

Findings

Cross-propagation improves label accuracy in each modality.

Method outperforms existing semi-supervised approaches on multiple datasets.

Effective with small amounts of labeled data.

Abstract

Technological advances facilitate the ability to acquire multimodal data, posing a challenge for recognition systems while also providing an opportunity to use the heterogeneous nature of the information to increase the generalization capability of models. An often overlooked issue is the cost of the labeling process, which is typically high due to the need for a significant investment in time and money associated with human experts. Existing semi-supervised learning methods often focus on operating in the feature space created by the fusion of available modalities, neglecting the potential for cross-utilizing complementary information available in each modality. To address this problem, we propose Cross-Modality Clustering-based Self-Labeling (CMCSL). Based on a small set of pre-labeled data, CMCSL groups instances belonging to each modality in the deep feature space and then propagates known labels within the resulting clusters. Next, information about the instances' class membership in each modality is exchanged based on the Euclidean distance to ensure more accurate labeling. Experimental evaluation conducted on 20 datasets derived from the MM-IMDb dataset indicates that cross-propagation of labels between modalities -- especially when the number of pre-labeled instances is small -- can allow for more reliable labeling and thus increase the classification performance in each modality.