Uncertainty-aware Active Learning of NeRF-based Object Models for Robot Manipulators using Visual and Re-orientation Actions

TL;DR

This paper introduces a method for robots to quickly learn complete 3D object models using NeRFs, active exploration, and re-orientation actions, improving manipulation success in unseen orientations.

Contribution

It presents an uncertainty-aware active learning framework that combines visual and re-orientation actions to enhance 3D object modeling for robotic manipulation.

Findings

14% improvement in visual reconstruction quality (PSNR)

20% enhancement in geometric surface reconstruction (F-score)

71% increase in manipulation success rate for unseen object orientations

Abstract

Manipulating unseen objects is challenging without a 3D representation, as objects generally have occluded surfaces. This requires physical interaction with objects to build their internal representations. This paper presents an approach that enables a robot to rapidly learn the complete 3D model of a given object for manipulation in unfamiliar orientations. We use an ensemble of partially constructed NeRF models to quantify model uncertainty to determine the next action (a visual or re-orientation action) by optimizing informativeness and feasibility. Further, our approach determines when and how to grasp and re-orient an object given its partial NeRF model and re-estimates the object pose to rectify misalignments introduced during the interaction. Experiments with a simulated Franka Emika Robot Manipulator operating in a tabletop environment with benchmark objects demonstrate an improvement of (i) 14% in visual reconstruction quality (PSNR), (ii) 20% in the geometric/depth reconstruction of the object surface (F-score) and (iii) 71% in the task success rate of manipulating objects a-priori unseen orientations/stable configurations in the scene; over current methods. The project page can be found here: https://actnerf.github.io.