Skeleton-based Action Recognition through Contrasting Two-Stream Spatial-Temporal Networks

TL;DR

This paper introduces a novel parallel two-stream network combining GCN and Transformer for skeleton-based action recognition, utilizing contrastive learning and a cyclical focal loss to enhance feature representation and achieve state-of-the-art results.

Contribution

It proposes a contrastive GCN-Transformer network with parallel streams and a cyclical focal loss, improving action recognition by capturing diverse features and focusing on hard samples.

Findings

Achieves state-of-the-art accuracy on benchmark datasets.

Effectively captures both local topology and global joint relationships.

Enhances feature learning through contrastive paradigm and cyclical focal loss.

Abstract

For pursuing accurate skeleton-based action recognition, most prior methods use the strategy of combining Graph Convolution Networks (GCNs) with attention-based methods in a serial way. However, they regard the human skeleton as a complete graph, resulting in less variations between different actions (e.g., the connection between the elbow and head in action ``clapping hands''). For this, we propose a novel Contrastive GCN-Transformer Network (ConGT) which fuses the spatial and temporal modules in a parallel way. The ConGT involves two parallel streams: Spatial-Temporal Graph Convolution stream (STG) and Spatial-Temporal Transformer stream (STT). The STG is designed to obtain action representations maintaining the natural topology structure of the human skeleton. The STT is devised to acquire action representations containing the global relationships among joints. Since the action representations produced from these two streams contain different characteristics, and each of them knows little information of the other, we introduce the contrastive learning paradigm to guide their output representations of the same sample to be as close as possible in a self-supervised manner. Through the contrastive learning, they can learn information from each other to enrich the action features by maximizing the mutual information between the two types of action representations. To further improve action recognition accuracy, we introduce the Cyclical Focal Loss (CFL) which can focus on confident training samples in early training epochs, with an increasing focus on hard samples during the middle epochs. We conduct experiments on three benchmark datasets, which demonstrate that our model achieves state-of-the-art performance in action recognition.