Med-SORA: Symptom to Organ Reasoning in Abdomen CT Images

TL;DR

Med-SORA is a novel framework that enhances symptom-to-organ reasoning in abdominal CT images by incorporating 3D features, soft labeling, and a cross-attention architecture, improving clinical interpretability and accuracy.

Contribution

It introduces a new symptom-to-organ reasoning framework with RAG-based dataset construction, soft labeling with organ anchors, and 2D-3D cross-attention, addressing limitations of prior models.

Findings

Outperforms existing models in symptom-organ reasoning accuracy.

Enables effective 3D clinical reasoning from CT images.

First to address symptom-to-organ reasoning with multimodal learning.

Abstract

Understanding symptom-image associations is crucial for clinical reasoning. However, existing medical multimodal models often rely on simple one-to-one hard labeling, oversimplifying clinical reality where symptoms relate to multiple organs. In addition, they mainly use single-slice 2D features without incorporating 3D information, limiting their ability to capture full anatomical context. In this study, we propose Med-SORA, a framework for symptom-to-organ reasoning in abdominal CT images. Med-SORA introduces RAG-based dataset construction, soft labeling with learnable organ anchors to capture one-to-many symptom-organ relationships, and a 2D-3D cross-attention architecture to fuse local and global image features. To our knowledge, this is the first work to address symptom-to-organ reasoning in medical multimodal learning. Experimental results show that Med-SORA outperforms existing medical multimodal models and enables accurate 3D clinical reasoning.