VR-Robo: A Real-to-Sim-to-Real Framework for Visual Robot Navigation and Locomotion

TL;DR

This paper introduces VR-Robo, a framework that creates photorealistic simulation environments for visual robot navigation and locomotion, enabling effective sim-to-real transfer and rapid adaptation in complex real-world settings.

Contribution

The paper presents a novel Real-to-Sim-to-Real framework using 3D Gaussian Splatting for scene reconstruction, supporting RGB perception and physical interactions for robot learning.

Findings

Successful RGB-only sim-to-real policy transfer

Enhanced robot policy adaptation in complex environments

Effective visual goal-tracking in real-world tests

Abstract

Recent success in legged robot locomotion is attributed to the integration of reinforcement learning and physical simulators. However, these policies often encounter challenges when deployed in real-world environments due to sim-to-real gaps, as simulators typically fail to replicate visual realism and complex real-world geometry. Moreover, the lack of realistic visual rendering limits the ability of these policies to support high-level tasks requiring RGB-based perception like ego-centric navigation. This paper presents a Real-to-Sim-to-Real framework that generates photorealistic and physically interactive "digital twin" simulation environments for visual navigation and locomotion learning. Our approach leverages 3D Gaussian Splatting (3DGS) based scene reconstruction from multi-view images and integrates these environments into simulations that support ego-centric visual perception and mesh-based physical interactions. To demonstrate its effectiveness, we train a reinforcement learning policy within the simulator to perform a visual goal-tracking task. Extensive experiments show that our framework achieves RGB-only sim-to-real policy transfer. Additionally, our framework facilitates the rapid adaptation of robot policies with effective exploration capability in complex new environments, highlighting its potential for applications in households and factories.