IMPACT-Scribe: Interactive Temporal Action Segmentation with Boundary Scribbles and Query Planning

TL;DR

IMPACT-Scribe is a correction-driven framework that improves dense temporal action annotation by leveraging human corrections to enhance future labeling efficiency and accuracy.

Contribution

It introduces a novel interactive approach combining uncertainty-aware supervision, query planning, and correction-driven adaptation for better human-machine collaboration.

Findings

Improves labeling quality per effort in dense temporal annotation.

Enhances boundary accuracy in action segmentation.

Fosters better human-machine interaction over time.

Abstract

Dense temporal annotation of procedural activity videos is vital for action understanding and embodied intelligence but remains labor-intensive due to reactive tools. Each correction is treated as an isolated edit, limiting reuse of information on annotator uncertainty and model reliability. We introduce IMPACT-Scribe, a correction-driven framework for dense labeling that uses each correction to improve future human-machine collaboration. IMPACT-Scribe combines uncertainty-aware boundary scribble supervision, local proposal modeling, cost-aware query planning, structured propagation, and correction-driven adaptation. Experiments and a human study show that this closed-loop design improves labeling quality per effort, enhances boundary accuracy, and fosters better human-machine interaction over time. The code will be made publicly available at https://github.com/BanzQians/IMPACT_AS.