NavigateDiff: Visual Predictors are Zero-Shot Navigation Assistants

TL;DR

NavigateDiff leverages vision-language models and diffusion networks to enable zero-shot visual navigation, improving robot adaptability and efficiency in unfamiliar environments without extensive retraining.

Contribution

The paper introduces NavigateDiff, a novel approach that combines large vision-language models with diffusion networks to predict future observations and guide zero-shot navigation.

Findings

Enhanced navigation robustness in diverse environments

Effective generalization to unseen scenes

Improved efficiency over traditional RL methods

Abstract

Navigating unfamiliar environments presents significant challenges for household robots, requiring the ability to recognize and reason about novel decoration and layout. Existing reinforcement learning methods cannot be directly transferred to new environments, as they typically rely on extensive mapping and exploration, leading to time-consuming and inefficient. To address these challenges, we try to transfer the logical knowledge and the generalization ability of pre-trained foundation models to zero-shot navigation. By integrating a large vision-language model with a diffusion network, our approach named \mname ~constructs a visual predictor that continuously predicts the agent's potential observations in the next step which can assist robots generate robust actions. Furthermore, to adapt the temporal property of navigation, we introduce temporal historical information to ensure that the predicted image is aligned with the navigation scene. We then carefully designed an information fusion framework that embeds the predicted future frames as guidance into goal-reaching policy to solve downstream image navigation tasks. This approach enhances navigation control and generalization across both simulated and real-world environments. Through extensive experimentation, we demonstrate the robustness and versatility of our method, showcasing its potential to improve the efficiency and effectiveness of robotic navigation in diverse settings.