VR-based generation of photorealistic synthetic data for training hand-object tracking models

TL;DR

This paper introduces blender-hoisynth, a VR-based interactive synthetic data generator for hand-object interaction training, producing photorealistic annotated videos that improve model training without performance loss.

Contribution

The paper presents a novel VR-enabled synthetic data generation tool that allows human-controlled, photorealistic HOI data creation for training 3D hand-object tracking models.

Findings

Photorealistic synthetic data can replace parts of real datasets without performance loss.

VR-based interaction enables precise control over hand-object interactions.

Generated data improves training efficiency for HOI models.

Abstract

Supervised learning models for precise tracking of hand-object interactions (HOI) in 3D require large amounts of annotated data for training. Moreover, it is not intuitive for non-experts to label 3D ground truth (e.g. 6DoF object pose) on 2D images. To address these issues, we present "blender-hoisynth", an interactive synthetic data generator based on the Blender software. Blender-hoisynth can scalably generate and automatically annotate visual HOI training data. Other competing approaches usually generate synthetic HOI data compeletely without human input. While this may be beneficial in some scenarios, HOI applications inherently necessitate direct control over the HOIs as an expression of human intent. With blender-hoisynth, it is possible for users to interact with objects via virtual hands using standard Virtual Reality hardware. The synthetically generated data are characterized by a high degree of photorealism and contain visually plausible and physically realistic videos of hands grasping objects and moving them around in 3D. To demonstrate the efficacy of our data generation, we replace large parts of the training data in the well-known DexYCB dataset with hoisynth data and train a state-of-the-art HOI reconstruction model with it. We show that there is no significant degradation in the model performance despite the data replacement.