Sign Segmentation with Changepoint-Modulated Pseudo-Labelling

TL;DR

This paper introduces a novel source-free domain adaptation method for sign language segmentation that leverages changepoint detection to improve pseudo-labeling, achieving state-of-the-art results across multiple datasets.

Contribution

It proposes the Changepoint-Modulated Pseudo-Labelling (CMPL) algorithm for domain adaptation in sign language segmentation, addressing the challenge of unlabelled target data.

Findings

Outperforms previous methods on BSL-1K and RWTH-PHOENIX-Weather datasets.

Effectively leverages motion cues for pseudo-labeling without target labels.

Demonstrates the viability of source-free domain adaptation in sign language segmentation.

Abstract

The objective of this work is to find temporal boundaries between signs in continuous sign language. Motivated by the paucity of annotation available for this task, we propose a simple yet effective algorithm to improve segmentation performance on unlabelled signing footage from a domain of interest. We make the following contributions: (1) We motivate and introduce the task of source-free domain adaptation for sign language segmentation, in which labelled source data is available for an initial training phase, but is not available during adaptation. (2) We propose the Changepoint-Modulated Pseudo-Labelling (CMPL) algorithm to leverage cues from abrupt changes in motion-sensitive feature space to improve pseudo-labelling quality for adaptation. (3) We showcase the effectiveness of our approach for category-agnostic sign segmentation, transferring from the BSLCORPUS to the BSL-1K and RWTH-PHOENIX-Weather 2014 datasets, where we outperform the prior state of the art.