HpEIS: Learning Hand Pose Embeddings for Multimedia Interactive Systems

TL;DR

HpEIS introduces a novel hand pose embedding system using VAE to enable stable, smooth, and visually interpretable mid-air hand interactions for multimedia applications with only a camera.

Contribution

The paper presents HpEIS, a new system that maps hand poses to a 2D space using VAE, with stability and smoothing improvements for better user interaction.

Findings

HpEIS provides stable and smooth hand pose interactions.

Users can effectively explore multimedia collections with HpEIS.

Experimental results show improved task completion times and accuracy.

Abstract

We present a novel Hand-pose Embedding Interactive System (HpEIS) as a virtual sensor, which maps users' flexible hand poses to a two-dimensional visual space using a Variational Autoencoder (VAE) trained on a variety of hand poses. HpEIS enables visually interpretable and guidable support for user explorations in multimedia collections, using only a camera as an external hand pose acquisition device. We identify general usability issues associated with system stability and smoothing requirements through pilot experiments with expert and inexperienced users. We then design stability and smoothing improvements, including hand-pose data augmentation, an anti-jitter regularisation term added to loss function, stabilising post-processing for movement turning points and smoothing post-processing based on One Euro Filters. In target selection experiments (n=12), we evaluate HpEIS by measures of task completion time and the final distance to target points, with and without the gesture guidance window condition. Experimental responses indicate that HpEIS provides users with a learnable, flexible, stable and smooth mid-air hand movement interaction experience.