Understanding Annotation Error Propagation and Learning an Adaptive Policy for Expert Intervention in Barrett's Video Segmentation

TL;DR

This paper investigates how annotation errors propagate in video segmentation and introduces an adaptive framework that optimally involves experts to improve accuracy while minimizing effort.

Contribution

It systematically analyzes error propagation in annotation prompts and proposes Learning-to-Re-Prompt (L2RP), a novel cost-aware method for expert intervention in video segmentation.

Findings

L2RP improves segmentation consistency and accuracy.

The method balances annotation effort with performance.

Experiments show superior results on Barrett's dysplasia and SUN-SEG datasets.

Abstract

Accurate annotation of endoscopic videos is essential yet time-consuming, particularly for challenging datasets such as dysplasia in Barrett's esophagus, where the affected regions are irregular and lack clear boundaries. Semi-automatic tools like Segment Anything Model 2 (SAM2) can ease this process by propagating annotations across frames, but small errors often accumulate and reduce accuracy, requiring expert review and correction. To address this, we systematically study how annotation errors propagate across different prompt types, namely masks, boxes, and points, and propose Learning-to-Re-Prompt (L2RP), a cost-aware framework that learns when and where to seek expert input. By tuning a human-cost parameter, our method balances annotation effort and segmentation accuracy. Experiments on a private Barrett's dysplasia dataset and the public SUN-SEG benchmark demonstrate improved temporal consistency and superior performance over baseline strategies.