# Dynamics Based 3D Skeletal Hand Tracking

## TL;DR

This paper introduces a novel real-time 3D hand tracking method using augmented rigid body simulation and multiple hypotheses, achieving robust accuracy with minimal hardware requirements.

## Contribution

It presents a new approach to hand tracking that models the hand as an articulated rigid body system solved via linear complementarity, improving robustness and efficiency.

## Key findings

- Real-time performance on a single CPU core
- High accuracy in challenging hand motions
- Robustness to noise and attachment errors

## Abstract

Tracking the full skeletal pose of the hands and fingers is a challenging problem that has a plethora of applications for user interaction. Existing techniques either require wearable hardware, add restrictions to user pose, or require significant computation resources. This research explores a new approach to tracking hands, or any articulated model, by using an augmented rigid body simulation. This allows us to phrase 3D object tracking as a linear complementarity problem with a well-defined solution. Based on a depth sensor's samples, the system generates constraints that limit motion orthogonal to the rigid body model's surface. These constraints, along with prior motion, collision/contact constraints, and joint mechanics, are resolved with a projected Gauss-Seidel solver. Due to camera noise properties and attachment errors, the numerous surface constraints are impulse capped to avoid overpowering mechanical constraints. To improve tracking accuracy, multiple simulations are spawned at each frame and fed a variety of heuristics, constraints and poses. A 3D error metric selects the best-fit simulation, helping the system handle challenging hand motions. Such an approach enables real-time, robust, and accurate 3D skeletal tracking of a user's hand on a variety of depth cameras, while only utilizing a single x86 CPU core for processing.