Zero Experience Required: Plug & Play Modular Transfer Learning for Semantic Visual Navigation

TL;DR

This paper introduces a modular transfer learning approach for visual navigation that enables zero-shot task generalization and significantly improves learning speed and performance across various navigation tasks and modalities.

Contribution

It presents a novel modular transfer learning model that leverages source task experience for multiple target tasks without task-specific training.

Findings

Outperforms state-of-the-art models on multiple datasets.

Enables zero-shot learning for new navigation tasks.

Learns faster and generalizes better across tasks.

Abstract

In reinforcement learning for visual navigation, it is common to develop a model for each new task, and train that model from scratch with task-specific interactions in 3D environments. However, this process is expensive; massive amounts of interactions are needed for the model to generalize well. Moreover, this process is repeated whenever there is a change in the task type or the goal modality. We present a unified approach to visual navigation using a novel modular transfer learning model. Our model can effectively leverage its experience from one source task and apply it to multiple target tasks (e.g., ObjectNav, RoomNav, ViewNav) with various goal modalities (e.g., image, sketch, audio, label). Furthermore, our model enables zero-shot experience learning, whereby it can solve the target tasks without receiving any task-specific interactive training. Our experiments on multiple photorealistic datasets and challenging tasks show that our approach learns faster, generalizes better, and outperforms SoTA models by a significant margin.