MammoWise: Multi-Model Local RAG Pipeline for Mammography Report Generation

TL;DR

MammoWise is a flexible, local multi-model pipeline that enhances mammography report generation and classification using open source vision language models, with improved accuracy through few-shot learning and retrieval augmentation.

Contribution

It introduces a versatile, privacy-preserving framework for mammogram reporting that supports multiple models, prompting strategies, and retrieval-augmented generation, with effective fine-tuning.

Findings

Report quality improves with few-shot prompting and RAG.

BI-RADS classification accuracy reaches 0.7545 after fine-tuning.

The pipeline is adaptable to various models and datasets.

Abstract

Screening mammography is high volume, time sensitive, and documentation heavy. Radiologists must translate subtle visual findings into consistent BI-RADS assessments, breast density categories, and structured narrative reports. While recent Vision Language Models (VLMs) enable image-to-text reporting, many rely on closed cloud systems or tightly coupled architectures that limit privacy, reproducibility, and adaptability. We present MammoWise, a local multi-model pipeline that transforms open source VLMs into mammogram report generators and multi-task classifiers. MammoWise supports any Ollama-hosted VLM and mammography dataset, and enables zero-shot, few-shot, and Chain-of-Thought prompting, with optional multimodal Retrieval Augmented Generation (RAG) using a vector database for case-specific context. We evaluate MedGemma, LLaVA-Med, and Qwen2.5-VL on VinDr-Mammo and DMID datasets, assessing report quality (BERTScore, ROUGE-L), BI-RADS classification, breast density, and key findings. Report generation is consistently strong and improves with few-shot prompting and RAG. Classification is feasible but sensitive to model and dataset choice. Parameter-efficient fine-tuning (QLoRA) of MedGemma improves reliability, achieving BI-RADS accuracy of 0.7545, density accuracy of 0.8840, and calcification accuracy of 0.9341 while preserving report quality. MammoWise provides a practical and extensible framework for deploying local VLMs for mammography reporting within a unified and reproducible workflow.