BUSTR: Breast Ultrasound Text Reporting with a Descriptor-Aware Vision-Language Model

TL;DR

BUSTR is a multitask vision-language model that generates breast ultrasound reports from images using structured descriptors and radiomics features, improving report quality and clinical relevance without needing paired datasets.

Contribution

It introduces a descriptor-aware vision-language framework trained with multitask and alignment losses, enabling report generation without paired image-report supervision.

Findings

Improves natural language generation metrics on two datasets.

Enhances clinical efficacy metrics, especially for BI-RADS and pathology.

Operates effectively without paired image-report data.

Abstract

Automated radiology report generation (RRG) for breast ultrasound (BUS) is limited by the lack of paired image-report datasets and the risk of hallucinations from large language models. We propose BUSTR, a multitask vision-language framework that generates BUS reports without requiring paired image-report supervision. BUSTR constructs reports from structured descriptors (e.g., BI-RADS, pathology, histology) and radiomics features, learns descriptor-aware visual representations with a multi-head Swin encoder trained using a multitask loss over dataset-specific descriptor sets, and aligns visual and textual tokens via a dual-level objective that combines token-level cross-entropy with a cosine-similarity alignment loss between input and output representations. We evaluate BUSTR on two public BUS datasets, BrEaST and BUS-BRA, which differ in size and available descriptors. Across both datasets, BUSTR consistently improves standard natural language generation metrics and clinical efficacy metrics, particularly for key targets such as BI-RADS category and pathology. Our results show that this descriptor-aware vision model, trained with a combined token-level and alignment loss, improves both automatic report metrics and clinical efficacy without requiring paired image-report data. The source code can be found at https://github.com/AAR-UNLV/BUSTR