Environment as Policy: Learning to Race in Unseen Tracks

TL;DR

This paper introduces an adaptive environment-shaping framework for reinforcement learning agents, enabling drone racing policies to generalize effectively to unseen tracks without retraining, by dynamically balancing environment difficulty during training.

Contribution

We propose a novel adaptive environment-shaping method using a secondary RL policy to improve generalization of drone racing agents to new tracks without retraining.

Findings

Agents successfully race in unseen challenging tracks

Method outperforms existing environment-shaping techniques

Effective in both simulation and real-world experiments

Abstract

Reinforcement learning (RL) has achieved outstanding success in complex robot control tasks, such as drone racing, where the RL agents have outperformed human champions in a known racing track. However, these agents fail in unseen track configurations, always requiring complete retraining when presented with new track layouts. This work aims to develop RL agents that generalize effectively to novel track configurations without retraining. The naive solution of training directly on a diverse set of track layouts can overburden the agent, resulting in suboptimal policy learning as the increased complexity of the environment impairs the agent's ability to learn to fly. To enhance the generalizability of the RL agent, we propose an adaptive environment-shaping framework that dynamically adjusts the training environment based on the agent's performance. We achieve this by leveraging a secondary RL policy to design environments that strike a balance between being challenging and achievable, allowing the agent to adapt and improve progressively. Using our adaptive environment shaping, one single racing policy efficiently learns to race in diverse challenging tracks. Experimental results validated in both simulation and the real world show that our method enables drones to successfully fly complex and unseen race tracks, outperforming existing environment-shaping techniques. Project page: http://rpg.ifi.uzh.ch/env_as_policy.