Disentangling Shared and Task-Specific Representations from Multi-Modal Clinical Data

TL;DR

This paper introduces OrthTD, a novel multi-task learning framework using a Transformer and orthogonal decomposition to effectively disentangle shared and task-specific features in multimodal clinical data, improving outcome prediction.

Contribution

The paper proposes OrthTD, a new method that separates shared and task-specific representations with orthogonality constraints, enhancing multi-outcome clinical predictions from multimodal data.

Findings

OrthTD achieved an average AUC of 87.5% and AUPRC of 37.2% on real-world data.

OrthTD outperformed existing tabular and multi-task methods in predictive accuracy.

Enforcing orthogonality reduced redundancy and improved detection of rare events.

Abstract

Real-world clinical data is inherently multimodal, providing complementary evidence that mirrors the practical necessity of jointly assessing multiple related outcomes. Although multi-task learning can improve efficiency by sharing information across outcomes, existing approaches often fail to balance shared representation learning with outcome-specific modeling. Hard parameter sharing can trigger negative transfer when task gradients conflict, while flexible sharing may still entangle shared and task-specific signals. To address this, we propose a multi-task framework built on a unified Transformer for multimodal fusion, augmented with Orthogonal Task Decomposition (OrthTD) to split patient representations into shared and task-specific subspaces and impose a geometric orthogonality constraint to reduce redundancy and isolate task-specific signals. We evaluated OrthTD on a real-world cohort of 12,430 surgical patients for predicting four outcomes. OrthTD achieved average AUC (area under the receiver operating characteristic curve) of 87.5% and average AUPRC (area under the precision-recall curve) of 37.2%, consistently outperformed advanced tabular and multi-task methods. Notably, OrthTD achieves substantial gains in AUPRC, indicating superior performance in identifying rare events within imbalanced clinical data. These results suggest that enforcing non-redundant shared and task-specific representations can improve multi-outcome prediction from multimodal clinical data.