An Evaluation of Hybrid Annotation Workflows on High-Ambiguity Spatiotemporal Video Footage

TL;DR

This paper evaluates a hybrid annotation workflow combining automatic pre-annotations with human verification for high-ambiguity spatiotemporal videos, demonstrating significant efficiency gains and providing a benchmarking framework.

Contribution

It introduces a novel framework for benchmarking AI-assisted annotation workflows, quantifies trade-offs, and shows a 35% reduction in annotation time using a tuned encoder in a human-in-the-loop process.

Findings

35% reduction in annotation time for most participants

Rigorous benchmarking framework for AI-assisted workflows

Demonstrated effectiveness on high-ambiguity spatiotemporal video data

Abstract

Manual annotation remains the gold standard for high-quality, dense temporal video datasets, yet it is inherently time-consuming. Vision-language models can aid human annotators and expedite this process. We report on the impact of automatic Pre-Annotations from a tuned encoder on a Human-in-the-Loop labeling workflow for video footage. Quantitative analysis in a study of a single-iteration test involving 18 volunteers demonstrates that our workflow reduced annotation time by 35% for the majority (72%) of the participants. Beyond efficiency, we provide a rigorous framework for benchmarking AI-assisted workflows that quantifies trade-offs between algorithmic speed and the integrity of human verification.