Exploring AI-based System Design for Pixel-level Protected Health Information Detection in Medical Images

TL;DR

This paper develops and evaluates an AI pipeline for pixel-level detection of protected health information in medical images, combining vision and language models to improve privacy protection in healthcare data sharing.

Contribution

It introduces a comprehensive AI-based pipeline with benchmarking of models for PHI detection, demonstrating the effectiveness of dedicated vision and language models for this task.

Findings

Optimal setup uses dedicated vision and language models for each module

LLMs improve OCR and enable end-to-end PHI detection

Achieves a good balance of performance, latency, and cost

Abstract

De-identification of medical images is a critical step to ensure privacy during data sharing in research and clinical settings. The initial step in this process involves detecting Protected Health Information (PHI), which can be found in image metadata or imprinted within image pixels. Despite the importance of such systems, there has been limited evaluation of existing AI-based solutions, creating barriers to the development of reliable and robust tools. In this study, we present an AI-based pipeline for PHI detection, comprising three key modules: text detection, text extraction, and text analysis. We benchmark three models - YOLOv11, EasyOCR, and GPT-4o - across different setups corresponding to these modules, evaluating their performance on two different datasets encompassing multiple imaging modalities and PHI categories. Our findings indicate that the optimal setup involves utilizing dedicated vision and language models for each module, which achieves a commendable balance in performance, latency, and cost associated with the usage of Large Language Models (LLMs). Additionally, we show that the application of LLMs not only involves identifying PHI content but also enhances OCR tasks and facilitates an end-to-end PHI detection pipeline, showcasing promising outcomes through our analysis.