DyTact: Capturing Dynamic Contacts in Hand-Object Manipulation

TL;DR

DyTact is a markerless, non-intrusive method for accurately capturing dynamic hand-object contacts in manipulation tasks, improving realism in animation, XR, and robotics.

Contribution

It introduces a novel dynamic surfel-based representation and adaptive sampling strategy for stable, fast, and accurate contact capture under occlusion.

Findings

Achieves state-of-the-art accuracy in dynamic contact estimation.

Enhances novel view synthesis quality significantly.

Operates with fast optimization and efficient memory usage.

Abstract

Reconstructing dynamic hand-object contacts is essential for realistic manipulation in AI character animation, XR, and robotics, yet it remains challenging due to heavy occlusions, complex surface details, and limitations in existing capture techniques. In this paper, we introduce DyTact, a markerless capture method for accurately capturing dynamic contact in hand-object manipulations in a non-intrusive manner. Our approach leverages a dynamic, articulated representation based on 2D Gaussian surfels to model complex manipulations. By binding these surfels to MANO meshes, DyTact harnesses the inductive bias of template models to stabilize and accelerate optimization. A refinement module addresses time-dependent high-frequency deformations, while a contact-guided adaptive sampling strategy selectively increases surfel density in contact regions to handle heavy occlusion. Extensive experiments demonstrate that DyTact not only achieves state-of-the-art dynamic contact estimation accuracy but also significantly improves novel view synthesis quality, all while operating with fast optimization and efficient memory usage. Project Page: https://oliver-cong02.github.io/DyTact.github.io/ .