Embodied Navigation with Auxiliary Task of Action Description Prediction

TL;DR

This paper introduces an auxiliary language description task into reinforcement learning for indoor robot navigation, enhancing explainability without sacrificing performance, and achieves state-of-the-art results in multimodal navigation.

Contribution

It proposes a novel method to incorporate action description prediction into reinforcement learning using knowledge distillation, improving explainability and performance in multimodal navigation.

Findings

Effective action description generation during navigation.

Achieved state-of-the-art in semantic audio-visual navigation.

Maintained high navigation performance with explainability.

Abstract

The field of multimodal robot navigation in indoor environments has garnered significant attention in recent years. However, as tasks and methods become more advanced, the action decision systems tend to become more complex and operate as black-boxes. For a reliable system, the ability to explain or describe its decisions is crucial; however, there tends to be a trade-off in that explainable systems can not outperform non-explainable systems in terms of performance. In this paper, we propose incorporating the task of describing actions in language into the reinforcement learning of navigation as an auxiliary task. Existing studies have found it difficult to incorporate describing actions into reinforcement learning due to the absence of ground-truth data. We address this issue by leveraging knowledge distillation from pre-trained description generation models, such as vision-language models. We comprehensively evaluate our approach across various navigation tasks, demonstrating that it can describe actions while attaining high navigation performance. Furthermore, it achieves state-of-the-art performance in the particularly challenging multimodal navigation task of semantic audio-visual navigation.