An unobtrusive quality supervision approach for medical image annotation

TL;DR

This paper proposes an unobtrusive system to evaluate medical image annotators' performance using synthetic cell images generated by advanced AI models, aiming to improve annotation quality in medical imaging.

Contribution

It introduces a novel approach combining generative models and user studies to unobtrusively assess annotator performance in medical image annotation.

Findings

Diffusion Models produce highly realistic synthetic cells.

Users failed to detect 52.12% of generated images.

Synthetic images are suitable for replacing original cells without detection.

Abstract

Image annotation is one essential prior step to enable data-driven algorithms. In medical imaging, having large and reliably annotated data sets is crucial to recognize various diseases robustly. However, annotator performance varies immensely, thus impacts model training. Therefore, often multiple annotators should be employed, which is however expensive and resource-intensive. Hence, it is desirable that users should annotate unseen data and have an automated system to unobtrusively rate their performance during this process. We examine such a system based on whole slide images (WSIs) showing lung fluid cells. We evaluate two methods the generation of synthetic individual cell images: conditional Generative Adversarial Networks and Diffusion Models (DM). For qualitative and quantitative evaluation, we conduct a user study to highlight the suitability of generated cells. Users could not detect 52.12% of generated images by DM proofing the feasibility to replace the original cells with synthetic cells without being noticed.