A Discordance-Aware Multimodal Framework with Multi-Agent Clinical Reasoning

TL;DR

This paper introduces a multimodal framework that models discordance between imaging and symptoms in knee osteoarthritis, enhancing clinical decision support with multi-agent reasoning and interpretable phenotypes.

Contribution

It presents a novel multi-agent system combining machine learning models and interpretability techniques to better understand and manage osteoarthritis discordance.

Findings

Models accurately predict joint space loss and pain progression.

The system computes a discordance score indicating symptom-structure mismatch.

Multi-agent reasoning provides clinically interpretable phenotypes and recommendations.

Abstract

Knee osteoarthritis frequently exhibits discordance between structural damage observed in imaging and patient-reported symptoms such as pain. This mismatch complicates clinical interpretation and patient stratification and remains insufficiently modeled in existing decision support systems. We propose a discordance aware multimodal framework that combines machine learning prediction models with a tool grounded multi agent reasoning system. Using baseline data from the FNIH Osteoarthritis Biomarkers Consortium, we trained multimodal models to predict two progression tasks, joint space loss only progression versus non progression, and pain only progression versus non progression. The predictive system integrates three modality specific experts: a CatBoost tabular model using demographic, radiographic, MRI-derived scalar, and biomarker features; MRI image embeddings extracted using a ResNet18 backbone; and Xray embeddings derived from the same architecture. Expert predictions are fused using a stacking ensemble. Residual based models estimate expected pain from structural features, enabling the computation of a pain structure discordance score between observed and expected symptoms. A multi-agent reasoning layer interprets these signals to assign clinically interpretable OA phenotypes and generate phenotype specific management recommendations.