Label Critic: Design Data Before Models

TL;DR

This paper introduces Label Critic, an automated tool that assesses AI-generated medical image labels through pairwise comparisons, significantly reducing manual annotation efforts and enabling high-quality dataset creation with minimal radiologist review.

Contribution

We developed Label Critic, an automatic label quality assessment tool that leverages large vision-language models to efficiently select and verify AI-generated annotations in medical imaging.

Findings

Achieves 96.5% accuracy in selecting the best label among pairs.

Reduces manual annotation time from 30-60 minutes to 15 seconds per scan.

Automatically discards low-quality AI labels with 81% accuracy.

Abstract

As medical datasets rapidly expand, creating detailed annotations of different body structures becomes increasingly expensive and time-consuming. We consider that requesting radiologists to create detailed annotations is unnecessarily burdensome and that pre-existing AI models can largely automate this process. Following the spirit don't use a sledgehammer on a nut, we find that, rather than creating annotations from scratch, radiologists only have to review and edit errors if the Best-AI Labels have mistakes. To obtain the Best-AI Labels among multiple AI Labels, we developed an automatic tool, called Label Critic, that can assess label quality through tireless pairwise comparisons. Extensive experiments demonstrate that, when incorporated with our developed Image-Prompt pairs, pre-existing Large Vision-Language Models (LVLM), trained on natural images and texts, achieve 96.5% accuracy when choosing the best label in a pair-wise comparison, without extra fine-tuning. By transforming the manual annotation task (30-60 min/scan) into an automatic comparison task (15 sec/scan), we effectively reduce the manual efforts required from radiologists by an order of magnitude. When the Best-AI Labels are sufficiently accurate (81% depending on body structures), they will be directly adopted as the gold-standard annotations for the dataset, with lower-quality AI Labels automatically discarded. Label Critic can also check the label quality of a single AI Label with 71.8% accuracy when no alternatives are available for comparison, prompting radiologists to review and edit if the estimated quality is low (19% depending on body structures).