GaitMA: Pose-guided Multi-modal Feature Fusion for Gait Recognition

TL;DR

GaitMA introduces a multi-modal gait recognition framework that combines silhouette and skeleton features using co-attention and mutual learning modules, achieving superior performance on multiple datasets.

Contribution

The paper proposes a novel multi-modal fusion approach with co-attention and mutual learning modules for robust gait recognition.

Findings

Outperforms existing methods on Gait3D, OU-MVLP, and CASIA-B datasets.

Effectively fuses silhouette and skeleton features for improved accuracy.

Demonstrates robustness against occlusions and semantic limitations.

Abstract

Gait recognition is a biometric technology that recognizes the identity of humans through their walking patterns. Existing appearance-based methods utilize CNN or Transformer to extract spatial and temporal features from silhouettes, while model-based methods employ GCN to focus on the special topological structure of skeleton points. However, the quality of silhouettes is limited by complex occlusions, and skeletons lack dense semantic features of the human body. To tackle these problems, we propose a novel gait recognition framework, dubbed Gait Multi-model Aggregation Network (GaitMA), which effectively combines two modalities to obtain a more robust and comprehensive gait representation for recognition. First, skeletons are represented by joint/limb-based heatmaps, and features from silhouettes and skeletons are respectively extracted using two CNN-based feature extractors. Second, a co-attention alignment module is proposed to align the features by element-wise attention. Finally, we propose a mutual learning module, which achieves feature fusion through cross-attention, Wasserstein loss is further introduced to ensure the effective fusion of two modalities. Extensive experimental results demonstrate the superiority of our model on Gait3D, OU-MVLP, and CASIA-B.