PETAR: Localized Findings Generation with Mask-Aware Vision-Language Modeling for PET Automated Reporting

TL;DR

This paper introduces PETAR, a novel 3D vision-language model and a large-scale dataset for automated PET report generation, addressing the complexity of 3D medical imaging and clinical findings with improved accuracy and clinical relevance.

Contribution

The paper presents PETAR-4B, a new 3D vision-language model for PET reporting, and PETARSeg-11K, the first large-scale dataset with lesion-level annotations for PET/CT imaging.

Findings

PETAR-4B outperforms existing 2D and 3D baselines in automated metrics.

A human study with physicians confirms the model's clinical utility.

The dataset enables detailed lesion-level analysis in PET/CT reports.

Abstract

Generating automated reports for 3D positron emission tomography (PET) is an important and challenging task in medical imaging. PET plays a vital role in oncology, but automating report generation is difficult due to the complexity of whole-body 3D volumes, the wide range of potential clinical findings, and the limited availability of annotated datasets. To address these challenges, we introduce PETARSeg-11K, the first large-scale, publicly available dataset that provides lesion-level correspondence between 3D PET/CT volumes and free-text radiological findings. It comprises 11,356 lesion descriptions paired with 3D segmentations. Second, we propose PETAR-4B, a 3D vision-language model designed for mask-aware, spatially grounded PET/CT reporting. PETAR-4B jointly encodes PET, CT, and 3D lesion segmentation masks, using a 3D focal prompt to capture fine-grained details of lesions that normally comprise less than 0.1% of the volume. Evaluations using automated metrics show PETAR-4B substantially outperforming all 2D and 3D baselines. A human study involving five physicians -- the first of its kind for automated PET reporting -- confirms the model's clinical utility and establishes correlations between automated metrics and expert judgment. This work provides a foundational dataset and a novel architecture, advancing 3D medical vision-language understanding in PET.