Camera-LiDAR Cross-modality Gait Recognition

TL;DR

This paper introduces CL-Gait, a novel cross-modality gait recognition framework utilizing camera and LiDAR data, employing a two-stream network and contrastive pre-training to address modality discrepancies and enable recognition across different sensor types.

Contribution

It is the first to propose a cross-modality gait recognition framework between camera and LiDAR, with a novel pre-training strategy using pseudo point clouds to align features.

Findings

Cross-modality gait recognition is feasible with the proposed model.

Contrastive pre-training reduces modality discrepancy.

The approach shows promising results despite inherent challenges.

Abstract

Gait recognition is a crucial biometric identification technique. Camera-based gait recognition has been widely applied in both research and industrial fields. LiDAR-based gait recognition has also begun to evolve most recently, due to the provision of 3D structural information. However, in certain applications, cameras fail to recognize persons, such as in low-light environments and long-distance recognition scenarios, where LiDARs work well. On the other hand, the deployment cost and complexity of LiDAR systems limit its wider application. Therefore, it is essential to consider cross-modality gait recognition between cameras and LiDARs for a broader range of applications. In this work, we propose the first cross-modality gait recognition framework between Camera and LiDAR, namely CL-Gait. It employs a two-stream network for feature embedding of both modalities. This poses a challenging recognition task due to the inherent matching between 3D and 2D data, exhibiting significant modality discrepancy. To align the feature spaces of the two modalities, i.e., camera silhouettes and LiDAR points, we propose a contrastive pre-training strategy to mitigate modality discrepancy. To make up for the absence of paired camera-LiDAR data for pre-training, we also introduce a strategy for generating data on a large scale. This strategy utilizes monocular depth estimated from single RGB images and virtual cameras to generate pseudo point clouds for contrastive pre-training. Extensive experiments show that the cross-modality gait recognition is very challenging but still contains potential and feasibility with our proposed model and pre-training strategy. To the best of our knowledge, this is the first work to address cross-modality gait recognition.