VTNet: Visual Transformer Network for Object Goal Navigation

TL;DR

This paper introduces VTNet, a visual transformer network that enhances object goal navigation by capturing relationships and spatial cues among objects, leading to improved navigation performance in unseen environments.

Contribution

The paper proposes a novel visual transformer architecture with a pre-training scheme that effectively encodes spatial and relational information for navigation tasks.

Findings

VTNet outperforms existing methods in AI2-Thor environments.

The spatial-aware descriptors improve navigation decision-making.

Pre-training enhances the association between visual features and navigation signals.

Abstract

Object goal navigation aims to steer an agent towards a target object based on observations of the agent. It is of pivotal importance to design effective visual representations of the observed scene in determining navigation actions. In this paper, we introduce a Visual Transformer Network (VTNet) for learning informative visual representation in navigation. VTNet is a highly effective structure that embodies two key properties for visual representations: First, the relationships among all the object instances in a scene are exploited; Second, the spatial locations of objects and image regions are emphasized so that directional navigation signals can be learned. Furthermore, we also develop a pre-training scheme to associate the visual representations with navigation signals, and thus facilitate navigation policy learning. In a nutshell, VTNet embeds object and region features with their location cues as spatial-aware descriptors and then incorporates all the encoded descriptors through attention operations to achieve informative representation for navigation. Given such visual representations, agents are able to explore the correlations between visual observations and navigation actions. For example, an agent would prioritize "turning right" over "turning left" when the visual representation emphasizes on the right side of activation map. Experiments in the artificial environment AI2-Thor demonstrate that VTNet significantly outperforms state-of-the-art methods in unseen testing environments.