PIGEON: VLM-Driven Object Navigation via Points of Interest Selection

TL;DR

PIGEON introduces a VLM-driven approach for object navigation that balances decision frequency and foresight by selecting Points of Interest, leading to state-of-the-art results in benchmark tests.

Contribution

The paper presents PIGEON, a novel method using semantic POI selection with VLMs for improved decision-making and zero-shot transfer in object navigation tasks.

Findings

Achieves state-of-the-art performance on object navigation benchmarks.

Enables deep reasoning during real-time navigation.

Improves decision frequency and semantic guidance through POI selection.

Abstract

Navigating to a specified object in an unknown environment is a fundamental yet challenging capability of embodied intelligence. However, current methods struggle to balance decision frequency with intelligence, resulting in decisions lacking foresight or discontinuous actions. In this work, we propose PIGEON: Point of Interest Guided Exploration for Object Navigation with VLM, maintaining a lightweight and semantically aligned snapshot memory during exploration as semantic input for the exploration strategy. We use a large Visual-Language Model (VLM), named PIGEON-VL, to select Points of Interest (PoI) formed during exploration and then employ a lower-level planner for action output, increasing the decision frequency. Additionally, this PoI-based decision-making enables the generation of Reinforcement Learning with Verifiable Reward (RLVR) data suitable for simulators. Experiments on classic object navigation benchmarks demonstrate that our zero-shot transfer method achieves state-of-the-art performance, while RLVR further enhances the model's semantic guidance capabilities, enabling deep reasoning during real-time navigation.