HOGSA: Bimanual Hand-Object Interaction Understanding with 3D Gaussian Splatting Based Data Augmentation

TL;DR

This paper introduces a novel 3D Gaussian Splatting data augmentation framework to generate large-scale, photorealistic bimanual hand-object interaction datasets, enhancing understanding in robotics and virtual reality.

Contribution

The work presents a new data augmentation method using 3D Gaussian Splatting and pose optimization to expand dataset diversity for bimanual hand-object interaction understanding.

Findings

Improved dataset diversity with various hand-object poses.

Enhanced baseline performance on H2O and Arctic benchmarks.

Effective augmentation reduces occlusion and pose variation issues.

Abstract

Understanding of bimanual hand-object interaction plays an important role in robotics and virtual reality. However, due to significant occlusions between hands and object as well as the high degree-of-freedom motions, it is challenging to collect and annotate a high-quality, large-scale dataset, which prevents further improvement of bimanual hand-object interaction-related baselines. In this work, we propose a new 3D Gaussian Splatting based data augmentation framework for bimanual hand-object interaction, which is capable of augmenting existing dataset to large-scale photorealistic data with various hand-object pose and viewpoints. First, we use mesh-based 3DGS to model objects and hands, and to deal with the rendering blur problem due to multi-resolution input images used, we design a super-resolution module. Second, we extend the single hand grasping pose optimization module for the bimanual hand object to generate various poses of bimanual hand-object interaction, which can significantly expand the pose distribution of the dataset. Third, we conduct an analysis for the impact of different aspects of the proposed data augmentation on the understanding of the bimanual hand-object interaction. We perform our data augmentation on two benchmarks, H2O and Arctic, and verify that our method can improve the performance of the baselines.