ExploreVLM: Closed-Loop Robot Exploration Task Planning with Vision-Language Models

TL;DR

ExploreVLM introduces a closed-loop, vision-language model-based framework for robot exploration that enables real-time plan adaptation and improved perception in dynamic environments.

Contribution

It presents a novel closed-loop task planning system with self-reflection and structured scene understanding, advancing interactive exploration capabilities of robots.

Findings

Outperforms state-of-the-art baselines in exploration tasks

Enables real-time plan adjustment through feedback mechanisms

Improves perception accuracy with structured scene representations

Abstract

The advancement of embodied intelligence is accelerating the integration of robots into daily life as human assistants. This evolution requires robots to not only interpret high-level instructions and plan tasks but also perceive and adapt within dynamic environments. Vision-Language Models (VLMs) present a promising solution by combining visual understanding and language reasoning. However, existing VLM-based methods struggle with interactive exploration, accurate perception, and real-time plan adaptation. To address these challenges, we propose ExploreVLM, a novel closed-loop task planning framework powered by Vision-Language Models (VLMs). The framework is built around a step-wise feedback mechanism that enables real-time plan adjustment and supports interactive exploration. At its core is a dual-stage task planner with self-reflection, enhanced by an object-centric spatial relation graph that provides structured, language-grounded scene representations to guide perception and planning. An execution validator supports the closed loop by verifying each action and triggering re-planning. Extensive real-world experiments demonstrate that ExploreVLM significantly outperforms state-of-the-art baselines, particularly in exploration-centric tasks. Ablation studies further validate the critical role of the reflective planner and structured perception in achieving robust and efficient task execution.