Clinically-Inspired Multi-Agent Transformers for Disease Trajectory Forecasting from Multimodal Data

TL;DR

This paper introduces a novel multi-agent transformer framework inspired by clinical decision-making to predict disease trajectories from multimodal data, outperforming existing methods in accuracy and calibration for conditions like osteoarthritis and Alzheimer's.

Contribution

We propose a clinically-inspired multi-agent transformer model that jointly analyzes imaging and clinical data for disease trajectory forecasting, with a new loss function and open-source implementation.

Findings

Outperforms state-of-the-art baselines in disease prediction accuracy

Effective in modeling temporal disease progression from multimodal data

Provides well-calibrated predictions suitable for clinical use

Abstract

Deep neural networks are often applied to medical images to automate the problem of medical diagnosis. However, a more clinically relevant question that practitioners usually face is how to predict the future trajectory of a disease. Current methods for prognosis or disease trajectory forecasting often require domain knowledge and are complicated to apply. In this paper, we formulate the prognosis prediction problem as a one-to-many prediction problem. Inspired by a clinical decision-making process with two agents -- a radiologist and a general practitioner -- we predict prognosis with two transformer-based components that share information with each other. The first transformer in this framework aims to analyze the imaging data, and the second one leverages its internal states as inputs, also fusing them with auxiliary clinical data. The temporal nature of the problem is modeled within the transformer states, allowing us to treat the forecasting problem as a multi-task classification, for which we propose a novel loss. We show the effectiveness of our approach in predicting the development of structural knee osteoarthritis changes and forecasting Alzheimer's disease clinical status directly from raw multi-modal data. The proposed method outperforms multiple state-of-the-art baselines with respect to performance and calibration, both of which are needed for real-world applications. An open-source implementation of our method is made publicly available at \url{https://github.com/Oulu-IMEDS/CLIMATv2}.