Timestamp-Supervised Action Segmentation from the Perspective of Clustering

TL;DR

This paper introduces a clustering-based framework for timestamp-supervised video action segmentation that improves pseudo-label quality by addressing ambiguous transition frames through ensembling and iterative clustering.

Contribution

It proposes a novel clustering perspective with pseudo-label ensembling and iterative clustering to enhance segmentation accuracy under timestamp supervision.

Findings

Effective pseudo-label refinement for ambiguous frames

Improved segmentation accuracy demonstrated in experiments

Clustering loss promotes intra-action feature compactness

Abstract

Video action segmentation under timestamp supervision has recently received much attention due to lower annotation costs. Most existing methods generate pseudo-labels for all frames in each video to train the segmentation model. However, these methods suffer from incorrect pseudo-labels, especially for the semantically unclear frames in the transition region between two consecutive actions, which we call ambiguous intervals. To address this issue, we propose a novel framework from the perspective of clustering, which includes the following two parts. First, pseudo-label ensembling generates incomplete but high-quality pseudo-label sequences, where the frames in ambiguous intervals have no pseudo-labels. Second, iterative clustering iteratively propagates the pseudo-labels to the ambiguous intervals by clustering, and thus updates the pseudo-label sequences to train the model. We further introduce a clustering loss, which encourages the features of frames within the same action segment more compact. Extensive experiments show the effectiveness of our method.