CLIMAT: Clinically-Inspired Multi-Agent Transformers for Knee Osteoarthritis Trajectory Forecasting

TL;DR

This paper introduces CLIMAT, a multi-agent transformer model inspired by clinical decision-making, to predict the progression of knee osteoarthritis using multimodal data, outperforming existing methods.

Contribution

The paper proposes a novel multi-agent transformer architecture that models clinical prognosis prediction by integrating imaging and auxiliary patient data.

Findings

Outperforms state-of-the-art baselines in osteoarthritis prognosis

Multi-agent transformers with depth 2 are sufficient for high performance

Effective integration of multimodal data improves prediction accuracy

Abstract

In medical applications, deep learning methods are built to automate diagnostic tasks. However, a clinically relevant question that practitioners usually face, is how to predict the future trajectory of a disease (prognosis). Current methods for such a problem often require domain knowledge, and are complicated to apply. In this paper, we formulate the prognosis prediction problem as a one-to-many forecasting problem from multimodal data. Inspired by a clinical decision-making process with two agents -- a radiologist and a general practitioner, we model a prognosis prediction problem with two transformer-based components that share information between each other. The first block in this model aims to analyze the imaging data, and the second block leverages the internal representations of the first one as inputs, also fusing them with auxiliary patient data. We show the effectiveness of our method in predicting the development of structural knee osteoarthritis changes over time. Our results show that the proposed method outperforms the state-of-the-art baselines in terms of various performance metrics. In addition, we empirically show that the existence of the multi-agent transformers with depths of 2 is sufficient to achieve good performances. Our code is publicly available at \url{https://github.com/MIPT-Oulu/CLIMAT}.