Gesture Recognition with Keypoint and Radar Stream Fusion for Automated Vehicles

TL;DR

This paper introduces a multi-modal gesture recognition system for autonomous vehicles using camera and radar data fusion, demonstrating improved accuracy and robustness under various conditions.

Contribution

It proposes a novel fusion neural network combining radar and camera streams with auxiliary losses, enhancing gesture recognition in autonomous driving scenarios.

Findings

Fusion improves gesture recognition accuracy.

System remains effective even when one sensor fails.

Multi-modal approach outperforms single modality methods.

Abstract

We present a joint camera and radar approach to enable autonomous vehicles to understand and react to human gestures in everyday traffic. Initially, we process the radar data with a PointNet followed by a spatio-temporal multilayer perceptron (stMLP). Independently, the human body pose is extracted from the camera frame and processed with a separate stMLP network. We propose a fusion neural network for both modalities, including an auxiliary loss for each modality. In our experiments with a collected dataset, we show the advantages of gesture recognition with two modalities. Motivated by adverse weather conditions, we also demonstrate promising performance when one of the sensors lacks functionality.