Context-Aware Sequence Alignment using 4D Skeletal Augmentation

TL;DR

This paper introduces CASA, a novel context-aware self-supervised learning framework that uses 4D skeletal augmentation and attention mechanisms to improve temporal alignment of human actions in videos, addressing discontinuity issues.

Contribution

CASA is the first to integrate 4D augmentation with attention mechanisms for sequence alignment, significantly enhancing alignment accuracy over prior methods.

Findings

CASA outperforms previous state-of-the-art methods on three public datasets.

It significantly improves phase progress and Kendall's Tau scores.

The approach effectively addresses temporal discontinuity in action sequences.

Abstract

Temporal alignment of fine-grained human actions in videos is important for numerous applications in computer vision, robotics, and mixed reality. State-of-the-art methods directly learn image-based embedding space by leveraging powerful deep convolutional neural networks. While being straightforward, their results are far from satisfactory, the aligned videos exhibit severe temporal discontinuity without additional post-processing steps. The recent advancements in human body and hand pose estimation in the wild promise new ways of addressing the task of human action alignment in videos. In this work, based on off-the-shelf human pose estimators, we propose a novel context-aware self-supervised learning architecture to align sequences of actions. We name it CASA. Specifically, CASA employs self-attention and cross-attention mechanisms to incorporate the spatial and temporal context of human actions, which can solve the temporal discontinuity problem. Moreover, we introduce a self-supervised learning scheme that is empowered by novel 4D augmentation techniques for 3D skeleton representations. We systematically evaluate the key components of our method. Our experiments on three public datasets demonstrate CASA significantly improves phase progress and Kendall's Tau scores over the previous state-of-the-art methods.