Self-supervised Extraction of Human Motion Structures via Frame-wise Discrete Features

TL;DR

This paper introduces a self-supervised encoder-decoder model that extracts sparse, frame-wise discrete features from human motion data, enabling visualization and effective recognition without human prior knowledge.

Contribution

It presents a novel method combining self-attention and vector clustering to extract and visualize human motion structures as discrete codes in a self-supervised manner.

Findings

Motion codes enable visualization of motion relationships.

Extracted codes achieve recognition performance comparable to task-specific methods.

The approach effectively captures sparse, shared motion features across sequences.

Abstract

The present paper proposes an encoder-decoder model for extracting the structures of human motions represented by frame-wise discrete features in a self-supervised manner. In the proposed method, features are extracted as codes in a motion codebook without the use of human knowledge, and the relationship between these codes can be visualized on a graph. Since the codes are expected to be temporally sparse compared to the captured frame rate and can be shared by multiple sequences, the proposed network model also addresses the need for training constraints. Specifically, the model consists of self-attention layers and a vector clustering block. The attention layers contribute to finding sparse keyframes and discrete features as motion codes, which are then extracted by vector clustering. The constraints are realized as training losses so that the same motion codes can be as contiguous as possible and can be shared by multiple sequences. In addition, we propose the use of causal self-attention as a method by which to calculate attention for long sequences consisting of numerous frames. In our experiments, the sparse structures of motion codes were used to compile a graph that facilitates visualization of the relationship between the codes and the differences between sequences. We then evaluated the effectiveness of the extracted motion codes by applying them to multiple recognition tasks and found that performance levels comparable to task-optimized methods could be achieved by linear probing.