The Radiance of Neural Fields: Democratizing Photorealistic and Dynamic Robotic Simulation

TL;DR

This paper introduces a novel neural rendering-based simulation platform that provides photorealistic, dynamic environments with animated human agents and realistic social interactions, advancing robotic navigation research.

Contribution

It presents the first neural rendering-based simulator combining photorealistic visuals, animated human agents, and social interaction modeling for robotic navigation tasks.

Findings

High-fidelity neural rendering of environments and humans.

Integration of social force models for realistic interactions.

Enables more realistic and accessible robotic simulation environments.

Abstract

As robots increasingly coexist with humans, they must navigate complex, dynamic environments rich in visual information and implicit social dynamics, like when to yield or move through crowds. Addressing these challenges requires significant advances in vision-based sensing and a deeper understanding of socio-dynamic factors, particularly in tasks like navigation. To facilitate this, robotics researchers need advanced simulation platforms offering dynamic, photorealistic environments with realistic actors. Unfortunately, most existing simulators fall short, prioritizing geometric accuracy over visual fidelity, and employing unrealistic agents with fixed trajectories and low-quality visuals. To overcome these limitations, we developed a simulator that incorporates three essential elements: (1) photorealistic neural rendering of environments, (2) neurally animated human entities with behavior management, and (3) an ego-centric robotic agent providing multi-sensor output. By utilizing advanced neural rendering techniques in a dual-NeRF simulator, our system produces high-fidelity, photorealistic renderings of both environments and human entities. Additionally, it integrates a state-of-the-art Social Force Model to model dynamic human-human and human-robot interactions, creating the first photorealistic and accessible human-robot simulation system powered by neural rendering.