A Generalized & Robust Framework For Timestamp Supervision in Temporal Action Segmentation

TL;DR

This paper introduces a robust EM-based framework for timestamp supervision in temporal action segmentation, effectively handling annotation errors and missing segments, and surpassing state-of-the-art results in various settings.

Contribution

The proposed method is the first to incorporate label uncertainty and robustness to annotation errors in timestamp supervision for action segmentation.

Findings

Achieves state-of-the-art results with accurate timestamp annotations.

Maintains stable performance with missing action segments.

Excels under the new Skip-tag supervision setup with random frame annotations.

Abstract

In temporal action segmentation, Timestamp supervision requires only a handful of labelled frames per video sequence. For unlabelled frames, previous works rely on assigning hard labels, and performance rapidly collapses under subtle violations of the annotation assumptions. We propose a novel Expectation-Maximization (EM) based approach that leverages the label uncertainty of unlabelled frames and is robust enough to accommodate possible annotation errors. With accurate timestamp annotations, our proposed method produces SOTA results and even exceeds the fully-supervised setup in several metrics and datasets. When applied to timestamp annotations with missing action segments, our method presents stable performance. To further test our formulation's robustness, we introduce the new challenging annotation setup of Skip-tag supervision. This setup relaxes constraints and requires annotations of any fixed number of random frames in a video, making it more flexible than Timestamp supervision while remaining competitive.