M3H: Multimodal Multitask Machine Learning for Healthcare

TL;DR

M3H is an explainable, scalable multimodal multitask machine learning framework that integrates diverse healthcare data types to improve multiple diagnostic and operational tasks, outperforming traditional models.

Contribution

The paper introduces M3H, a novel multimodal multitask learning framework with an innovative attention mechanism and explainability, applicable across various healthcare tasks and data types.

Findings

Outperforms traditional single-task models by 11.6% on average

Handles diverse data modalities including tabular, time-series, language, and vision

Applicable in resource-constrained hospital environments

Abstract

Developing an integrated many-to-many framework leveraging multimodal data for multiple tasks is crucial to unifying healthcare applications ranging from diagnoses to operations. In resource-constrained hospital environments, a scalable and unified machine learning framework that improves previous forecast performances could improve hospital operations and save costs. We introduce M3H, an explainable Multimodal Multitask Machine Learning for Healthcare framework that consolidates learning from tabular, time-series, language, and vision data for supervised binary/multiclass classification, regression, and unsupervised clustering. It features a novel attention mechanism balancing self-exploitation (learning source-task), and cross-exploration (learning cross-tasks), and offers explainability through a proposed TIM score, shedding light on the dynamics of task learning interdependencies. M3H encompasses an unprecedented range of medical tasks and machine learning problem classes and consistently outperforms traditional single-task models by on average 11.6% across 40 disease diagnoses from 16 medical departments, three hospital operation forecasts, and one patient phenotyping task. The modular design of the framework ensures its generalizability in data processing, task definition, and rapid model prototyping, making it production ready for both clinical and operational healthcare settings, especially those in constrained environments.