LARGO: Low-Rank Hypernetwork for Handling Missing Modalities

TL;DR

LARGO introduces a hypernetwork approach using tensor decomposition to efficiently handle missing modalities in multimodal image analysis, outperforming existing methods across multiple datasets.

Contribution

The paper presents LARGO, a novel hypernetwork that models weights with tensor decomposition to unify missing-modality models without architectural changes.

Findings

Ranks first in 47 out of 52 configurations on BraTS and ISLES datasets.

Achieves average Dice improvements of +0.68% and +2.53% over baselines.

Demonstrates potential extension to non-medical modalities.

Abstract

Addressing missing modalities is an important challenge in multimodal image analysis and often relies on complex architectures that do not transfer easily to different datasets without architectural modifications or hyperparameter tuning. While most existing methods tackle this problem in feature space by engineering representations that are robust to missing inputs, we instead operate in weight space. We propose LARGO, a hypernetwork that compresses the $2^N-1$ dedicated missing-modality models into a single network by modelling the convolutional weights using the Canonical Polyadic (CP) tensor decomposition. Extensive experimental validation on BraTS 2018 (4 modalities, 15 scenarios) and ISLES 2022 (3 modalities, 7 scenarios) shows that our method ranks first in 47 out of 52 configurations, achieving average Dice improvements of +0.68$\%$ and +2.53$\%$ over state-of-the-art baselines (mmFormer, M$^{3}$AE, ShaSpec, SimMLM). A proof-of-concept experiment on avMNIST suggests that LARGO may extend beyond medical imaging to heterogeneous non-medical modalities.