GRASPLAT: Enabling dexterous grasping through novel view synthesis

TL;DR

GRASPLAT introduces a novel RGB-based grasping framework that synthesizes realistic hand-object images using 3D Gaussian Splatting, significantly improving grasp success rates without requiring 3D scans.

Contribution

It presents a new approach leveraging novel view synthesis and photometric refinement for dexterous grasping from RGB images, bypassing the need for high-quality 3D data.

Findings

Achieves up to 36.9% higher grasp success rates.

Effectively synthesizes realistic hand-object views for training.

Outperforms existing image-based grasping methods.

Abstract

Achieving dexterous robotic grasping with multi-fingered hands remains a significant challenge. While existing methods rely on complete 3D scans to predict grasp poses, these approaches face limitations due to the difficulty of acquiring high-quality 3D data in real-world scenarios. In this paper, we introduce GRASPLAT, a novel grasping framework that leverages consistent 3D information while being trained solely on RGB images. Our key insight is that by synthesizing physically plausible images of a hand grasping an object, we can regress the corresponding hand joints for a successful grasp. To achieve this, we utilize 3D Gaussian Splatting to generate high-fidelity novel views of real hand-object interactions, enabling end-to-end training with RGB data. Unlike prior methods, our approach incorporates a photometric loss that refines grasp predictions by minimizing discrepancies between rendered and real images. We conduct extensive experiments on both synthetic and real-world grasping datasets, demonstrating that GRASPLAT improves grasp success rates up to 36.9% over existing image-based methods. Project page: https://mbortolon97.github.io/grasplat/