Segmentation, Detection and Explanation: A Unified Framework for CT Appearance Reasoning

TL;DR

This paper introduces a unified framework for CT image analysis that combines language-guided visual reasoning with progressive localization, improving segmentation and detection accuracy while providing interpretability.

Contribution

It proposes a novel autoregressive model integrating language and vision for comprehensive CT interpretation, with a new dataset and a coarse-to-fine attention mechanism.

Findings

Achieved up to 1.0% Dice improvement on BTCV benchmark.

Achieved up to 1.7% Dice improvement on MosMed+ benchmark.

Provided additional appearance reasoning outputs.

Abstract

Recent progress in deep learning has significantly advanced CT image analysis, particularly for segmentation tasks. However, these advances are largely confined to image-level pattern recognition, with most methods lacking explicit anatomical or contextual reasoning. Large vision-language models introduce linguistic context into image analysis, yet most approaches typically focus on a single task, which is insufficient for clinical workflow analysis that requires multiple fine-grained types of analysis, such as anatomy detection and segmentation. In this paper, we propose a unified autoregressive framework that integrates language-guided visual reasoning into CT interpretation. Our method introduces task-routing tokens that trigger detection and segmentation heads conditioned on the hidden states of a large vision-language model, enabling coherent generation of visual outputs (e.g., masks and bounding boxes) and textual reasonings. To progressively enhance localisation accuracy and semantic clarity, we further design a "closer-look" mechanism that allows the model to perform progressive coarse-to-fine visits to regions of interest under refined fields of view. To support model training and evaluation, we curated a new multimodal CT dataset containing pixel-wise masks, bounding boxes, spatial prompts, and structured descriptions for visual objects constructed through an AI-assisted annotation process with human verification. Experiments on public benchmarks demonstrate consistent improvements over the SoTA, achieving up to 1.0% Dice on BTCV and 1.7% Dice on MosMed+, while additionally providing appearance reasoning outputs. The code and dataset will be available.