Towards an End-To-End System for Real-Time Gesture Recognition from Surface Vibrations

TL;DR

This paper introduces an end-to-end gesture recognition system using surface vibration sensors on a desk, combining hardware and a configurable data pipeline with deep learning for accurate, unobtrusive interaction.

Contribution

It presents a custom low-noise vibration sensor system and a modular processing pipeline optimized with hyperparameter search for real-time gesture recognition.

Findings

Achieved high accuracy in gesture classification across various data splits.

Demonstrated strong user-independent performance in leave-one-subject-out validation.

Developed a low-noise piezoelectric sensor system for surface vibration sensing.

Abstract

Sensing surface vibrations promise unobtrusive interaction for smart home systems by enabling gesture recognition on existing everyday surfaces without disturbing living-space design. Existing approaches typically address only parts of the processing chain, such as sensing hardware or offline gesture recognition, rather than providing an end-to-end system from surface-mounted sensors to the evaluation of the prediction model. This paper presents a custom sensor system and a configurable data-to-model pipeline for gesture recognition on a standard office desk. Our hardware enables a low-noise sensing of the vibrations using piezoelectric sensors. Building on a modular signal-processing framework, we model the full chain from continuous recordings through variable pre-processing to a model-ready dataset, and process the resulting data with compact depthwise separable 1D-CNNs. We conduct a joint search over pre-processing and model hyperparameters and identify a configuration with 8,722 parameters that uses band-pass filtering, fixed-length windows, and min-max normalization. On a self-recorded dataset with 15 participants performing six gestures this configuration achieves high accuracies across different data splitting methods, including strong user-independent performance in a leave-one-subject-out cross-validation.