From Seeing to Experiencing: Scaling Navigation Foundation Models with Reinforcement Learning

TL;DR

This paper introduces the Seeing-to-Experiencing framework that enhances navigation foundation models with reinforcement learning, improving their interactivity, safety, and generalization in real-world urban environments through novel training strategies and comprehensive benchmarking.

Contribution

It presents a new framework combining pretraining on videos with RL-based post-training, along with innovative strategies and a benchmark for evaluating navigation models in photorealistic environments.

Findings

RL improves safety and interactivity over offline training alone.

The Anchor-Guided Distribution Matching stabilizes learning and models diverse motions.

RL-based post-training outperforms supervised fine-tuning in robot navigation tasks.

Abstract

Navigation foundation models trained on massive webscale data enable agents to generalize across diverse environments and embodiments. However, these models trained solely on offline data, often lack the capacity to reason about the consequences of their actions or adapt through counterfactual understanding. They thus face significant limitations in the real-world urban navigation where interactive and safe behaviors, such as avoiding obstacles and moving pedestrians, are critical. To tackle these challenges, we introduce the Seeing-to-Experiencing framework to scale the capability of navigation foundation models with reinforcement learning. S2E combines the strengths of pre-training on videos and post-training through RL. It maintains the generalizability acquired from large-scale real-world videos while enhancing its interactivity through RL in simulation environments. Specifically, we introduce two innovations: an Anchor-Guided Distribution Matching strategy, which stabilizes learning and models diverse motion patterns through anchor-based supervision; and a Residual-Attention Module, which obtains reactive behaviors from simulation environments without erasing the model's pretrained knowledge. Moreover, we establish a comprehensive end-to-end evaluation benchmark, NavBench-GS, built on photorealistic 3DGS reconstructions of real-world scenes that incorporate physical interactions. It can systematically assess the generalizability and safety of navigation foundation models. Extensive experiments show that S2E mitigates the diminishing returns often seen when scaling with offline data alone. We perform a thorough analysis of the benefits of Reinforcement Learning compared to Supervised Fine-Tuning in the context of post-training for robot learning. Our findings emphasize the crucial role of integrating interactive online experiences to effectively scale foundation models in Robotics.