PrefPaint: Enhancing Medical Image Inpainting through Expert Human Feedback

TL;DR

PrefPaint is an interactive system that integrates expert human feedback into medical image inpainting, improving the realism and anatomical accuracy of generated images for clinical AI, while being resource-efficient.

Contribution

It introduces PrefPaint, a novel human-in-the-loop inpainting framework using D3PO, a web interface, and Model Tree versioning for effective expert feedback and model management.

Findings

PrefPaint reduces visual inconsistencies in generated images.

The system produces more realistic, anatomically accurate polyp images.

User studies confirm PrefPaint's superiority over existing methods.

Abstract

Inpainting, the process of filling missing or corrupted image parts, has broad applications in medical imaging. However, generating anatomically accurate synthetic polyp images for clinical AI is a largely underexplored problem. In specialized fields like gastroenterology, inaccuracies in generated images can lead to false patterns and significant errors in downstream diagnosis. To ensure reliability, models require direct feedback from domain experts like oncologists. We propose PrefPaint, an interactive system that incorporates expert human feedback into Stable Diffusion Inpainting. By using D3PO instead of full RLHF, our approach bypasses the need for computationally expensive reward models, making it a highly practical choice for resource-constrained clinical settings. Furthermore, we introduce a streamlined web-based interface to facilitate this expert-in-the-loop training. Central to this platform is the Model Tree versioning interface, a novel HCI concept that visualizes the evolutionary progression of fine-tuned models. This interactive interface provides a smooth and intuitive user experience, making it easier to offer feedback and manage the fine-tuning process. User studies show that PrefPaint outperforms existing methods, reducing visual inconsistencies and generating highly realistic, anatomically accurate polyp images suitable for clinical AI applications.