Exploiting Radiance Fields for Grasp Generation on Novel Synthetic Views

TL;DR

This paper demonstrates that synthesizing novel views using radiance fields can enhance robot grasp planning by providing additional context, leading to more accurate and comprehensive grasp pose generation without additional real-world image captures.

Contribution

The work shows initial evidence that radiance field-based novel view synthesis improves grasp pose generation in robotic manipulation tasks.

Findings

Novel views contribute additional force-closure grasps.

Improved grasp coverage with synthesized views.

Enhanced grasp quality from virtual viewpoint augmentation.

Abstract

Vision based robot manipulation uses cameras to capture one or more images of a scene containing the objects to be manipulated. Taking multiple images can help if any object is occluded from one viewpoint but more visible from another viewpoint. However, the camera has to be moved to a sequence of suitable positions for capturing multiple images, which requires time and may not always be possible, due to reachability constraints. So while additional images can produce more accurate grasp poses due to the extra information available, the time-cost goes up with the number of additional views sampled. Scene representations like Gaussian Splatting are capable of rendering accurate photorealistic virtual images from user-specified novel viewpoints. In this work, we show initial results which indicate that novel view synthesis can provide additional context in generating grasp poses. Our experiments on the Graspnet-1billion dataset show that novel views contributed force-closure grasps in addition to the force-closure grasps obtained from sparsely sampled real views while also improving grasp coverage. In the future we hope this work can be extended to improve grasp extraction from radiance fields constructed with a single input image, using for example diffusion models or generalizable radiance fields.