# Vision-based Navigation with Language-based Assistance via Imitation Learning with Indirect Intervention

## TL;DR

This paper introduces a vision-language navigation task where an agent is guided by language and can query an advisor when lost, using a novel imitation learning framework to improve success in indoor environments.

## Contribution

The paper proposes a new grounded vision-language navigation task and a general imitation learning framework with indirect intervention for effective guidance.

## Key findings

- Significant success rate improvements over baselines.
- Effective in both seen and unseen environments.
- Code and data are publicly available.

## Abstract

We present Vision-based Navigation with Language-based Assistance (VNLA), a grounded vision-language task where an agent with visual perception is guided via language to find objects in photorealistic indoor environments. The task emulates a real-world scenario in that (a) the requester may not know how to navigate to the target objects and thus makes requests by only specifying high-level end-goals, and (b) the agent is capable of sensing when it is lost and querying an advisor, who is more qualified at the task, to obtain language subgoals to make progress. To model language-based assistance, we develop a general framework termed Imitation Learning with Indirect Intervention (I3L), and propose a solution that is effective on the VNLA task. Empirical results show that this approach significantly improves the success rate of the learning agent over other baselines in both seen and unseen environments. Our code and data are publicly available at https://github.com/debadeepta/vnla .