SMC-NCA: Semantic-guided Multi-level Contrast for Semi-supervised Temporal Action Segmentation

TL;DR

This paper introduces SMC-NCA, a novel semi-supervised learning framework that leverages semantic-guided multi-level contrast and neighborhood consistency to improve temporal action segmentation in videos with limited labels.

Contribution

The paper proposes a new contrastive learning scheme with a neighborhood consistency unit for better frame representation in semi-supervised temporal action segmentation.

Findings

SMC-NCA outperforms state-of-the-art methods on three benchmarks.

Significant improvements in Edit distance and accuracy with limited labeled data.

Effective generalization demonstrated on Parkinson's Disease Mouse Behaviour dataset.

Abstract

Semi-supervised temporal action segmentation (SS-TA) aims to perform frame-wise classification in long untrimmed videos, where only a fraction of videos in the training set have labels. Recent studies have shown the potential of contrastive learning in unsupervised representation learning using unlabelled data. However, learning the representation of each frame by unsupervised contrastive learning for action segmentation remains an open and challenging problem. In this paper, we propose a novel Semantic-guided Multi-level Contrast scheme with a Neighbourhood-Consistency-Aware unit (SMC-NCA) to extract strong frame-wise representations for SS-TAS. Specifically, for representation learning, SMC is first used to explore intra- and inter-information variations in a unified and contrastive way, based on action-specific semantic information and temporal information highlighting relations between actions. Then, the NCA module, which is responsible for enforcing spatial consistency between neighbourhoods centered at different frames to alleviate over-segmentation issues, works alongside SMC for semi-supervised learning (SSL). Our SMC outperforms the other state-of-the-art methods on three benchmarks, offering improvements of up to 17.8% and 12.6% in terms of Edit distance and accuracy, respectively. Additionally, the NCA unit results in significantly better segmentation performance in the presence of only 5% labelled videos. We also demonstrate the generalizability and effectiveness of the proposed method on our Parkinson Disease's Mouse Behaviour (PDMB) dataset. Code is available at https://github.com/FeixiangZhou/SMC-NCA.