NeRF2Real: Sim2real Transfer of Vision-guided Bipedal Motion Skills using Neural Radiance Fields

TL;DR

This paper introduces a novel sim2real transfer method for vision-guided bipedal robots using Neural Radiance Fields to model static scenes from short videos, enabling realistic simulation and successful policy transfer to real robots.

Contribution

It proposes a new approach combining NeRF-based scene modeling with physics simulation for effective sim2real transfer of vision-based robot policies.

Findings

Successful transfer of navigation and ball-pushing policies to real robots

Realistic scene modeling from short videos using NeRF

Effective simulation of contact dynamics for policy learning

Abstract

We present a system for applying sim2real approaches to "in the wild" scenes with realistic visuals, and to policies which rely on active perception using RGB cameras. Given a short video of a static scene collected using a generic phone, we learn the scene's contact geometry and a function for novel view synthesis using a Neural Radiance Field (NeRF). We augment the NeRF rendering of the static scene by overlaying the rendering of other dynamic objects (e.g. the robot's own body, a ball). A simulation is then created using the rendering engine in a physics simulator which computes contact dynamics from the static scene geometry (estimated from the NeRF volume density) and the dynamic objects' geometry and physical properties (assumed known). We demonstrate that we can use this simulation to learn vision-based whole body navigation and ball pushing policies for a 20 degrees of freedom humanoid robot with an actuated head-mounted RGB camera, and we successfully transfer these policies to a real robot. Project video is available at https://sites.google.com/view/nerf2real/home