Learning to Map Natural Language Instructions to Physical Quadcopter Control using Simulated Flight

TL;DR

This paper introduces a novel framework that combines simulation and real-world data to enable a quadcopter to follow natural language instructions and explore environments effectively without requiring physical flight during training.

Contribution

The paper presents SuReAL, a joint simulation and real-world learning method that maps natural language instructions to quadcopter control, integrating supervised and reinforcement learning without physical flight during training.

Findings

Effective instruction-following demonstrated on physical quadcopters.

Model predicts exploration needs and visit likelihoods.

Framework enables learning without autonomous flight in real environments.

Abstract

We propose a joint simulation and real-world learning framework for mapping navigation instructions and raw first-person observations to continuous control. Our model estimates the need for environment exploration, predicts the likelihood of visiting environment positions during execution, and controls the agent to both explore and visit high-likelihood positions. We introduce Supervised Reinforcement Asynchronous Learning (SuReAL). Learning uses both simulation and real environments without requiring autonomous flight in the physical environment during training, and combines supervised learning for predicting positions to visit and reinforcement learning for continuous control. We evaluate our approach on a natural language instruction-following task with a physical quadcopter, and demonstrate effective execution and exploration behavior.