Spatially-Aware Speaker for Vision-and-Language Navigation Instruction Generation

TL;DR

This paper introduces SAS, a spatially-aware speaker model that generates detailed, diverse navigation instructions for embodied robots by leveraging environmental knowledge and adversarial training to improve instruction quality.

Contribution

The paper presents SAS, a novel instruction generator that incorporates structural and semantic environment knowledge and uses adversarial reward learning to enhance instruction diversity and quality.

Findings

Outperforms existing models on standard metrics

Produces more detailed and varied navigation instructions

Utilizes environment knowledge for instruction generation

Abstract

Embodied AI aims to develop robots that can \textit{understand} and execute human language instructions, as well as communicate in natural languages. On this front, we study the task of generating highly detailed navigational instructions for the embodied robots to follow. Although recent studies have demonstrated significant leaps in the generation of step-by-step instructions from sequences of images, the generated instructions lack variety in terms of their referral to objects and landmarks. Existing speaker models learn strategies to evade the evaluation metrics and obtain higher scores even for low-quality sentences. In this work, we propose SAS (Spatially-Aware Speaker), an instruction generator or \textit{Speaker} model that utilises both structural and semantic knowledge of the environment to produce richer instructions. For training, we employ a reward learning method in an adversarial setting to avoid systematic bias introduced by language evaluation metrics. Empirically, our method outperforms existing instruction generation models, evaluated using standard metrics. Our code is available at \url{https://github.com/gmuraleekrishna/SAS}.