Forward and Inverse models in HCI:Physical simulation and deep learning for inferring 3D finger pose

TL;DR

This paper explores forward and inverse modeling approaches using machine learning and simulation to accurately infer 3D finger position and pose on mobile devices with capacitive sensors, enhancing HCI interactions.

Contribution

It introduces a data-driven framework combining deep learning and electrostatic simulation to improve finger pose inference accuracy in HCI systems.

Findings

Machine learning models accelerate electrostatic simulation by a factor of millions.

Combining forward and inverse models yields the most accurate finger pose inference.

Data from robots, simulators, and humans improve model robustness.

Abstract

We outline the role of forward and inverse modelling approaches in the design of human--computer interaction systems. Causal, forward models tend to be easier to specify and simulate, but HCI requires solutions of the inverse problem. We infer finger 3D position $(x,y,z)$ and pose (pitch and yaw) on a mobile device using capacitive sensors which can sense the finger up to 5cm above the screen. We use machine learning to develop data-driven models to infer position, pose and sensor readings, based on training data from: 1. data generated by robots, 2. data from electrostatic simulators 3. human-generated data. Machine learned emulation is used to accelerate the electrostatic simulation performance by a factor of millions. We combine a Conditional Variational Autoencoder with domain expertise/models experimentally collected data. We compare forward and inverse model approaches to direct inference of finger pose. The combination gives the most accurate reported results on inferring 3D position and pose with a capacitive sensor on a mobile device.