Controllable Navigation Instruction Generation with Chain of Thought Prompting

TL;DR

This paper introduces C-Instructor, a novel method leveraging chain-of-thought prompting with landmarks and spatial modeling to generate controllable navigation instructions, outperforming previous models in quality and controllability.

Contribution

The paper presents a new approach using chain-of-thought prompts, landmark guidance, and spatial topology modeling for style- and content-controllable instruction generation.

Findings

C-Instructor outperforms previous methods in text metrics

Generated instructions are more accessible and controllable

User studies favor C-Instructor's instructions

Abstract

Instruction generation is a vital and multidisciplinary research area with broad applications. Existing instruction generation models are limited to generating instructions in a single style from a particular dataset, and the style and content of generated instructions cannot be controlled. Moreover, most existing instruction generation methods also disregard the spatial modeling of the navigation environment. Leveraging the capabilities of Large Language Models (LLMs), we propose C-Instructor, which utilizes the chain-of-thought-style prompt for style-controllable and content-controllable instruction generation. Firstly, we propose a Chain of Thought with Landmarks (CoTL) mechanism, which guides the LLM to identify key landmarks and then generate complete instructions. CoTL renders generated instructions more accessible to follow and offers greater controllability over the manipulation of landmark objects. Furthermore, we present a Spatial Topology Modeling Task to facilitate the understanding of the spatial structure of the environment. Finally, we introduce a Style-Mixed Training policy, harnessing the prior knowledge of LLMs to enable style control for instruction generation based on different prompts within a single model instance. Extensive experiments demonstrate that instructions generated by C-Instructor outperform those generated by previous methods in text metrics, navigation guidance evaluation, and user studies.