Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning

TL;DR

This paper introduces 'Help, Anna!', a simulator where an agent learns to request and interpret natural language and visual assistance to improve object-finding tasks, demonstrating better help-seeking behavior and success rates.

Contribution

The paper presents a hierarchical, memory-augmented neural agent trained with imitation learning to effectively seek help in complex visual navigation tasks.

Findings

Agent asks for help more effectively than baselines

Achieves higher success rates in seen and unseen environments

Demonstrates the effectiveness of retrospective curiosity in imitation learning

Abstract

Mobile agents that can leverage help from humans can potentially accomplish more complex tasks than they could entirely on their own. We develop "Help, Anna!" (HANNA), an interactive photo-realistic simulator in which an agent fulfills object-finding tasks by requesting and interpreting natural language-and-vision assistance. An agent solving tasks in a HANNA environment can leverage simulated human assistants, called ANNA (Automatic Natural Navigation Assistants), which, upon request, provide natural language and visual instructions to direct the agent towards the goals. To address the HANNA problem, we develop a memory-augmented neural agent that hierarchically models multiple levels of decision-making, and an imitation learning algorithm that teaches the agent to avoid repeating past mistakes while simultaneously predicting its own chances of making future progress. Empirically, our approach is able to ask for help more effectively than competitive baselines and, thus, attains higher task success rate on both previously seen and previously unseen environments. We publicly release code and data at https://github.com/khanhptnk/hanna . A video demo is available at https://youtu.be/18P94aaaLKg .