VISITRON: Visual Semantics-Aligned Interactively Trained Object-Navigator

TL;DR

VISITRON is a multi-modal Transformer-based robot navigator designed for interactive vision-and-dialog tasks, effectively leveraging dialogue and visual semantics to improve navigation and interaction decisions in photo-realistic environments.

Contribution

It introduces a novel Transformer-based model that aligns visual semantics with dialogue, and learns when to interact versus navigate, advancing interactive robot navigation.

Findings

Achieves state-of-the-art SPL performance on CVDN benchmark.

Effectively identifies when to interact, enabling better generalization.

Competitive results on static CVDN leaderboard.

Abstract

Interactive robots navigating photo-realistic environments need to be trained to effectively leverage and handle the dynamic nature of dialogue in addition to the challenges underlying vision-and-language navigation (VLN). In this paper, we present VISITRON, a multi-modal Transformer-based navigator better suited to the interactive regime inherent to Cooperative Vision-and-Dialog Navigation (CVDN). VISITRON is trained to: i) identify and associate object-level concepts and semantics between the environment and dialogue history, ii) identify when to interact vs. navigate via imitation learning of a binary classification head. We perform extensive pre-training and fine-tuning ablations with VISITRON to gain empirical insights and improve performance on CVDN. VISITRON's ability to identify when to interact leads to a natural generalization of the game-play mode introduced by Roman et al. (arXiv:2005.00728) for enabling the use of such models in different environments. VISITRON is competitive with models on the static CVDN leaderboard and attains state-of-the-art performance on the Success weighted by Path Length (SPL) metric.