Zero-shot Object Navigation with Vision-Language Models Reasoning

TL;DR

This paper introduces VLTNet, a novel vision-language model with Tree-of-Thought reasoning for zero-shot object navigation, enabling robots to follow natural language instructions in unknown environments without prior training.

Contribution

The paper presents a new VLTNet model that integrates Tree-of-Thought reasoning for improved zero-shot object navigation using natural language instructions.

Findings

VLTNet outperforms existing methods on PASTURE and RoboTHOR benchmarks.

Tree-of-Thought reasoning improves navigation accuracy in complex scenarios.

The model effectively handles natural language instructions for object navigation.

Abstract

Object navigation is crucial for robots, but traditional methods require substantial training data and cannot be generalized to unknown environments. Zero-shot object navigation (ZSON) aims to address this challenge, allowing robots to interact with unknown objects without specific training data. Language-driven zero-shot object navigation (L-ZSON) is an extension of ZSON that incorporates natural language instructions to guide robot navigation and interaction with objects. In this paper, we propose a novel Vision Language model with a Tree-of-thought Network (VLTNet) for L-ZSON. VLTNet comprises four main modules: vision language model understanding, semantic mapping, tree-of-thought reasoning and exploration, and goal identification. Among these modules, Tree-of-Thought (ToT) reasoning and exploration module serves as a core component, innovatively using the ToT reasoning framework for navigation frontier selection during robot exploration. Compared to conventional frontier selection without reasoning, navigation using ToT reasoning involves multi-path reasoning processes and backtracking when necessary, enabling globally informed decision-making with higher accuracy. Experimental results on PASTURE and RoboTHOR benchmarks demonstrate the outstanding performance of our model in LZSON, particularly in scenarios involving complex natural language as target instructions.