Multimodal Large Language Model for Visual Navigation

TL;DR

This paper introduces a fine-tuned multimodal large language model for visual navigation that simplifies prompts and improves performance, reducing collisions compared to existing methods.

Contribution

The work presents a novel approach to visual navigation by fine-tuning large language models with a simple prompt and history collection, avoiding complex prompt engineering.

Findings

Outperforms state-of-the-art behavior cloning methods

Reduces collision rates in visual navigation tasks

Effective use of human demonstrations and collision signals

Abstract

Recent efforts to enable visual navigation using large language models have mainly focused on developing complex prompt systems. These systems incorporate instructions, observations, and history into massive text prompts, which are then combined with pre-trained large language models to facilitate visual navigation. In contrast, our approach aims to fine-tune large language models for visual navigation without extensive prompt engineering. Our design involves a simple text prompt, current observations, and a history collector model that gathers information from previous observations as input. For output, our design provides a probability distribution of possible actions that the agent can take during navigation. We train our model using human demonstrations and collision signals from the Habitat-Matterport 3D Dataset (HM3D). Experimental results demonstrate that our method outperforms state-of-the-art behavior cloning methods and effectively reduces collision rates.