Jointly Learning Structured Representations and Stabilized Affinity for Human Motion Segmentation

TL;DR

This paper introduces TDSC, a novel method for human motion segmentation that jointly learns structured representations and stabilized affinity to improve accuracy and robustness, especially when raw features violate traditional assumptions.

Contribution

The proposed TDSC approach innovatively combines self-expressive modeling with temporal constraints and a stabilization mechanism for affinity, advancing HMS performance.

Findings

TDSC outperforms existing methods on five benchmark datasets.

It effectively handles raw features that violate traditional subspace assumptions.

The approach is validated with both conventional and deep features.

Abstract

Human Motion Segmentation (HMS), which aims to partition a video into non-overlapping segments corresponding to different human motions, has recently attracted increasing research attention. Existing HMS approaches are predominantly based on subspace clustering, which are grounded on the assumption that the distribution of high-dimensional temporal features well aligns with a Union-of-Subspaces (UoS). For videos in the real world, however, the raw frame-level features often violate the UoS assumption and yield unsatisfactory segmentation performance. To address this issue, we propose an efficient and effective approach for HMS, named Temporal Deep Self-expressive subspace Clustering (TDSC), which jointly learns temporally consistent structured representations and stabilized affinity for accurate and robust HMS. Specifically, in TDSC, we alternately learn structured representations of the input frame features and self-expressive coefficients via a properly regularized self-expressive model, in which a coding-rate maximization regularizer is incorporated to avoid representation collapse and conform the learned representations to span a desired UoS distribution, and meanwhile, temporal constraints are incorporated to promote temporally adjacent frames to be partitioned into the same groups. Moreover, we develop a temporal momentum averaging mechanism to stabilize affinity evolution and design a reparameterization strategy to enable efficient optimization. We conduct extensive experiments on five benchmark HMS datasets using both conventional (HoG) and up-to-date deep features (i.e., CLIP, DINOv2) to validate the effectiveness of our approach.