CT-Agent: A Multimodal-LLM Agent for 3D CT Radiology Question Answering

TL;DR

This paper introduces CT-Agent, a multimodal framework that enhances 3D CT radiology question answering by addressing anatomical complexity and spatial relationships, outperforming existing systems on two datasets.

Contribution

The paper presents a novel multimodal agentic framework that effectively handles 3D CT data complexity and spatial relationships for improved radiology question answering.

Findings

Superior performance on CT-RATE dataset

Effective handling of anatomical complexity

Accurate spatial relationship capture

Abstract

Computed Tomography (CT) scan, which produces 3D volumetric medical data that can be viewed as hundreds of cross-sectional images (a.k.a. slices), provides detailed anatomical information for diagnosis. For radiologists, creating CT radiology reports is time-consuming and error-prone. A visual question answering (VQA) system that can answer radiologists' questions about some anatomical regions on the CT scan and even automatically generate a radiology report is urgently needed. However, existing VQA systems cannot adequately handle the CT radiology question answering (CTQA) task for: (1) anatomic complexity makes CT images difficult to understand; (2) spatial relationship across hundreds slices is difficult to capture. To address these issues, this paper proposes CT-Agent, a multimodal agentic framework for CTQA. CT-Agent adopts anatomically independent tools to break down the anatomic complexity; furthermore, it efficiently captures the across-slice spatial relationship with a global-local token compression strategy. Experimental results on two 3D chest CT datasets, CT-RATE and RadGenome-ChestCT, verify the superior performance of CT-Agent.