MedRegion-CT: Region-Focused Multimodal LLM for Comprehensive 3D CT Report Generation

TL;DR

MedRegion-CT is a region-focused multimodal LLM that enhances 3D CT report generation by capturing region-specific details, improving clinical relevance and interpretability over existing global-feature methods.

Contribution

We introduce a novel region-focused multimodal LLM framework with region pooling, pseudo-mask processing, and patient-specific attributions for improved 3D CT report generation.

Findings

Achieved state-of-the-art performance on RadGenome-Chest CT report generation benchmark.

Outperformed existing methods in natural language quality and clinical relevance.

Enhanced interpretability through patient-specific attributions.

Abstract

The recent release of RadGenome-Chest CT has significantly advanced CT-based report generation. However, existing methods primarily focus on global features, making it challenging to capture region-specific details, which may cause certain abnormalities to go unnoticed. To address this, we propose MedRegion-CT, a region-focused Multi-Modal Large Language Model (MLLM) framework, featuring three key innovations. First, we introduce Region Representative ($R^2$) Token Pooling, which utilizes a 2D-wise pretrained vision model to efficiently extract 3D CT features. This approach generates global tokens representing overall slice features and region tokens highlighting target areas, enabling the MLLM to process comprehensive information effectively. Second, a universal segmentation model generates pseudo-masks, which are then processed by a mask encoder to extract region-centric features. This allows the MLLM to focus on clinically relevant regions, using six predefined region masks. Third, we leverage segmentation results to extract patient-specific attributions, including organ size, diameter, and locations. These are converted into text prompts, enriching the MLLM's understanding of patient-specific contexts. To ensure rigorous evaluation, we conducted benchmark experiments on report generation using the RadGenome-Chest CT. MedRegion-CT achieved state-of-the-art performance, outperforming existing methods in natural language generation quality and clinical relevance while maintaining interpretability. The code for our framework is publicly available.