Together, Then Apart: Revisiting Multimodal Survival Analysis via a Min-Max Perspective

TL;DR

This paper introduces TTA, a novel min-max framework for multimodal survival analysis that balances alignment and modality-specific features, leading to improved performance and interpretability.

Contribution

It proposes a unified approach that jointly models shared and unique modality representations using a min-max optimization, addressing overemphasis on alignment in prior methods.

Findings

TTA outperforms state-of-the-art methods on five TCGA benchmarks.

The framework enhances representational diversity and interpretability.

Theoretical insights into balancing alignment and distinctiveness in multimodal analysis.

Abstract

Integrating heterogeneous modalities such as histopathology and genomics is central to advancing survival analysis, yet most existing methods prioritize cross-modal alignment through attention-based fusion mechanisms, often at the expense of modality-specific characteristics. This overemphasis on alignment leads to representation collapse and reduced diversity. In this work, we revisit multi-modal survival analysis via the dual lens of alignment and distinctiveness, positing that preserving modality-specific structure is as vital as achieving semantic coherence. In this paper, we introduce Together-Then-Apart (TTA), a unified min-max optimization framework that simultaneously models shared and modality-specific representations. The Together stage minimizes semantic discrepancies by aligning embeddings via shared prototypes, guided by an unbalanced optimal transport objective that adaptively highlights informative tokens. The Apart stage maximizes representational diversity through modality anchors and a contrastive regularizer that preserve unique modality information and prevent feature collapse. Extensive experiments on five TCGA benchmarks show that TTA consistently outperforms state-of-the-art methods. Beyond empirical gains, our formulation provides a new theoretical perspective of how alignment and distinctiveness can be jointly achieved in for robust, interpretable, and biologically meaningful multi-modal survival analysis.