Multimodal feature fusion for CNN-based gait recognition: an empirical comparison

TL;DR

This paper compares CNN architectures and multimodal fusion methods for gait recognition using raw pixels, optical flow, and depth maps, demonstrating that simple inputs and effective fusion can achieve state-of-the-art results.

Contribution

It provides a comprehensive empirical comparison of different CNN modalities and fusion strategies for gait recognition, highlighting the effectiveness of raw pixel inputs and multimodal fusion.

Findings

Raw pixel inputs are competitive with silhouette-based features.

Fusion of multiple modalities improves recognition accuracy.

Proper CNN architecture design is crucial for optimal performance.

Abstract

People identification in video based on the way they walk (i.e. gait) is a relevant task in computer vision using a non-invasive approach. Standard and current approaches typically derive gait signatures from sequences of binary energy maps of subjects extracted from images, but this process introduces a large amount of non-stationary noise, thus, conditioning their efficacy. In contrast, in this paper we focus on the raw pixels, or simple functions derived from them, letting advanced learning techniques to extract relevant features. Therefore, we present a comparative study of different Convolutional Neural Network (CNN) architectures by using three different modalities (i.e. gray pixels, optical flow channels and depth maps) on two widely-adopted and challenging datasets: TUM-GAID and CASIA-B. In addition, we perform a comparative study between different early and late fusion methods used to combine the information obtained from each kind of modalities. Our experimental results suggest that (i) the raw pixel values represent a competitive input modality, compared to the traditional state-of-the-art silhouette-based features (e.g. GEI), since equivalent or better results are obtained; (ii) the fusion of the raw pixel information with information from optical flow and depth maps allows to obtain state-of-the-art results on the gait recognition task with an image resolution several times smaller than the previously reported results; and, (iii) the selection and the design of the CNN architecture are critical points that can make a difference between state-of-the-art results or poor ones.