Hierarchical Vector Quantization for Unsupervised Action Segmentation

TL;DR

This paper introduces Hierarchical Vector Quantization (HVQ), a novel method for unsupervised temporal action segmentation that effectively captures intra-class variations and improves segmentation quality across multiple datasets.

Contribution

The paper proposes HVQ, a hierarchical clustering approach with two vector quantization modules, addressing intra-class variation issues in unsupervised action segmentation.

Findings

HVQ outperforms state-of-the-art methods in F1 score and recall.

The new Jensen-Shannon Distance metric effectively evaluates segmentation quality.

HVQ better captures segment length distribution than existing approaches.

Abstract

In this work, we address unsupervised temporal action segmentation, which segments a set of long, untrimmed videos into semantically meaningful segments that are consistent across videos. While recent approaches combine representation learning and clustering in a single step for this task, they do not cope with large variations within temporal segments of the same class. To address this limitation, we propose a novel method, termed Hierarchical Vector Quantization (HVQ), that consists of two subsequent vector quantization modules. This results in a hierarchical clustering where the additional subclusters cover the variations within a cluster. We demonstrate that our approach captures the distribution of segment lengths much better than the state of the art. To this end, we introduce a new metric based on the Jensen-Shannon Distance (JSD) for unsupervised temporal action segmentation. We evaluate our approach on three public datasets, namely Breakfast, YouTube Instructional and IKEA ASM. Our approach outperforms the state of the art in terms of F1 score, recall and JSD.