Towards Selection of Large Multimodal Models as Engines for Burned-in Protected Health Information Detection in Medical Images

TL;DR

This study benchmarks large multimodal models for detecting protected health information in medical images, highlighting their OCR capabilities and providing practical deployment recommendations.

Contribution

It systematically compares LMMs like GPT-4o, Gemini 2.5, and Qwen 2.5 7B for PHI detection, offering insights into their performance and deployment strategies.

Findings

LMMs outperform traditional OCR in OCR accuracy.

Improved OCR does not always lead to better PHI detection.

Best results occur with complex imprint patterns and strong LMMs.

Abstract

The detection of Protected Health Information (PHI) in medical imaging is critical for safeguarding patient privacy and ensuring compliance with regulatory frameworks. Traditional detection methodologies predominantly utilize Optical Character Recognition (OCR) models in conjunction with named entity recognition. However, recent advancements in Large Multimodal Model (LMM) present new opportunities for enhanced text extraction and semantic analysis. In this study, we systematically benchmark three prominent closed and open-sourced LMMs, namely GPT-4o, Gemini 2.5 Flash, and Qwen 2.5 7B, utilizing two distinct pipeline configurations: one dedicated to text analysis alone and another integrating both OCR and semantic analysis. Our results indicate that LMM exhibits superior OCR efficacy (WER: 0.03-0.05, CER: 0.02-0.03) compared to conventional models like EasyOCR. However, this improvement in OCR performance does not consistently correlate with enhanced overall PHI detection accuracy. The strongest performance gains are observed on test cases with complex imprint patterns. In scenarios where text regions are well readable with sufficient contrast, and strong LMMs are employed for text analysis after OCR, different pipeline configurations yield similar results. Furthermore, we provide empirically grounded recommendations for LMM selection tailored to specific operational constraints and propose a deployment strategy that leverages scalable and modular infrastructure.