Ske2Grid: Skeleton-to-Grid Representation Learning for Action Recognition

TL;DR

Ske2Grid introduces a novel grid-based skeleton representation with a learnable convolution framework, significantly improving skeleton-based action recognition accuracy across multiple datasets.

Contribution

The paper proposes Ske2Grid, a new grid representation learning method with a graph-node index transform, up-sampling transform, and progressive learning strategy for enhanced action recognition.

Findings

Outperforms existing GCN-based methods on six datasets

Achieves significant accuracy improvements without complex techniques

Demonstrates the effectiveness of grid-based skeleton representations

Abstract

This paper presents Ske2Grid, a new representation learning framework for improved skeleton-based action recognition. In Ske2Grid, we define a regular convolution operation upon a novel grid representation of human skeleton, which is a compact image-like grid patch constructed and learned through three novel designs. Specifically, we propose a graph-node index transform (GIT) to construct a regular grid patch through assigning the nodes in the skeleton graph one by one to the desired grid cells. To ensure that GIT is a bijection and enrich the expressiveness of the grid representation, an up-sampling transform (UPT) is learned to interpolate the skeleton graph nodes for filling the grid patch to the full. To resolve the problem when the one-step UPT is aggressive and further exploit the representation capability of the grid patch with increasing spatial size, a progressive learning strategy (PLS) is proposed which decouples the UPT into multiple steps and aligns them to multiple paired GITs through a compact cascaded design learned progressively. We construct networks upon prevailing graph convolution networks and conduct experiments on six mainstream skeleton-based action recognition datasets. Experiments show that our Ske2Grid significantly outperforms existing GCN-based solutions under different benchmark settings, without bells and whistles. Code and models are available at https://github.com/OSVAI/Ske2Grid