Inference-Time Dynamic Modality Selection for Incomplete Multimodal Classification

TL;DR

DyMo is an inference-time framework that adaptively selects and fuses reliable modalities in incomplete multimodal data, improving classification performance without discarding or blindly imputing missing information.

Contribution

The paper introduces DyMo, a novel dynamic modality selection method that maximizes task-relevant information at inference time, with a new selection algorithm and a theoretical information-loss connection.

Findings

DyMo outperforms state-of-the-art methods across multiple datasets.

The approach effectively handles various missing-data scenarios.

The method improves classification accuracy in incomplete multimodal settings.

Abstract

Multimodal deep learning (MDL) has achieved remarkable success across various domains, yet its practical deployment is often hindered by incomplete multimodal data. Existing incomplete MDL methods either discard missing modalities, risking the loss of valuable task-relevant information, or recover them, potentially introducing irrelevant noise, leading to the discarding-imputation dilemma. To address this dilemma, in this paper, we propose DyMo, a new inference-time dynamic modality selection framework that adaptively identifies and fuses reliable recovered modalities, fully exploring task-relevant information beyond the conventional discard-or-impute paradigm. Central to DyMo is a novel selection algorithm that maximizes multimodal task-relevant information for each test sample. Since direct estimation of such information at test time is intractable due to the unknown data distribution, we theoretically establish a connection between information and the task loss, which we compute at inference time as a tractable proxy. Building on this, a novel principled reward function is proposed to guide modality selection. In addition, we design a flexible multimodal network architecture compatible with arbitrary modality combinations, alongside a tailored training strategy for robust representation learning. Extensive experiments on diverse natural and medical image datasets show that DyMo significantly outperforms state-of-the-art incomplete/dynamic MDL methods across various missing-data scenarios. Our code is available at https://github.com//siyi-wind/DyMo.