Log2Motion: Biomechanical Motion Synthesis from Touch Logs

TL;DR

Log2Motion introduces a reinforcement learning-based biomechanical simulation method to generate realistic motion sequences from touch logs, enhancing understanding of motor control and ergonomics in mobile device interactions.

Contribution

This work presents the first approach to synthesize biomechanically plausible motion directly from touch logs using reinforcement learning and physics simulation.

Findings

Generated motions closely match human motion data

The method provides detailed estimates of movement parameters

Demonstrated effectiveness on large-scale touch datasets

Abstract

Touch data from mobile devices are collected at scale but reveal little about the interactions that produce them. While biomechanical simulations can illuminate motor control processes, they have not yet been developed for touch interactions. To close this gap, we propose a novel computational problem: synthesizing plausible motion directly from logs. Our key insight is a reinforcement learning-driven musculoskeletal forward simulation that generates biomechanically plausible motion sequences consistent with events recorded in touch logs. We achieve this by integrating a software emulator into a physics simulator, allowing biomechanical models to manipulate real applications in real-time. Log2Motion produces rich syntheses of user movements from touch logs, including estimates of motion, speed, accuracy, and effort. We assess the plausibility of generated movements by comparing against human data from a motion capture study and prior findings, and demonstrate Log2Motion in a large-scale dataset. Biomechanical motion synthesis provides a new way to understand log data, illuminating the ergonomics and motor control underlying touch interactions.