3D Whole-body Grasp Synthesis with Directional Controllability

TL;DR

This paper introduces CWGrasp, a novel method for synthesizing realistic 3D whole-body grasps with directional controllability, addressing data scarcity and improving plausibility and efficiency in grasp synthesis.

Contribution

CWGrasp is the first approach to incorporate early geometry-based reasoning for controllable, plausible 3D whole-body grasp synthesis, handling both hands and receptacle awareness.

Findings

Outperforms baselines on GRAB and ReplicaGrasp datasets

Achieves lower runtime and computational budget

Effectively models both right and left-hand grasps

Abstract

Synthesizing 3D whole bodies that realistically grasp objects is useful for animation, mixed reality, and robotics. This is challenging, because the hands and body need to look natural w.r.t. each other, the grasped object, as well as the local scene (i.e., a receptacle supporting the object). Moreover, training data for this task is really scarce, while capturing new data is expensive. Recent work goes beyond finite datasets via a divide-and-conquer approach; it first generates a "guiding" right-hand grasp, and then searches for bodies that match this. However, the guiding-hand synthesis lacks controllability and receptacle awareness, so it likely has an implausible direction (i.e., a body can't match this without penetrating the receptacle) and needs corrections through major post-processing. Moreover, the body search needs exhaustive sampling and is expensive. These are strong limitations. We tackle these with a novel method called CWGrasp. Our key idea is that performing geometry-based reasoning "early on," instead of "too late," provides rich "control" signals for inference. To this end, CWGrasp first samples a plausible reaching-direction vector (used later for both the arm and hand) from a probabilistic model built via ray-casting from the object and collision checking. Moreover, CWGrasp uniquely tackles both right and left-hand grasps. We evaluate on the GRAB and ReplicaGrasp datasets. CWGrasp outperforms baselines, at lower runtime and budget, while all components help performance. Code and models are available at https://gpaschalidis.github.io/cwgrasp.