Gesture Recognition with mmWave Wi-Fi Access Points: Lessons Learned

TL;DR

This paper explores using mmWave Wi-Fi signals, specifically beam SNRs and CSI, for gesture recognition, demonstrating high accuracy with deep learning and analyzing robustness across environments.

Contribution

It introduces a novel approach using mmWave beam SNRs for gesture recognition and compares their robustness to CSI across different environments.

Findings

Deep neural network achieves 96.7% accuracy with beam SNRs.

CSI features generalize across environments without re-training.

Beam SNR features require re-training for different environments.

Abstract

In recent years, channel state information (CSI) at sub-6 GHz has been widely exploited for Wi-Fi sensing, particularly for activity and gesture recognition. In this work, we instead explore mmWave (60 GHz) Wi-Fi signals for gesture recognition/pose estimation. Our focus is on the mmWave Wi-Fi signals so that they can be used not only for high data rate communication but also for improved sensing e.g., for extended reality (XR) applications. For this reason, we extract spatial beam signal-to-noise ratios (SNRs) from the periodic beam training employed by IEEE 802.11ad devices. We consider a set of 10 gestures/poses motivated by XR applications. We conduct experiments in two environments and with three people.As a comparison, we also collect CSI from IEEE 802.11ac devices. To extract features from the CSI and the beam SNR, we leverage a deep neural network (DNN). The DNN classifier achieves promising results on the beam SNR task with state-of-the-art 96.7% accuracy in a single environment, even with a limited dataset. We also investigate the robustness of the beam SNR against CSI across different environments. Our experiments reveal that features from the CSI generalize without additional re-training, while those from beam SNRs do not. Therefore, re-training is required in the latter case.