Beyond Fixed Tasks: Unsupervised Environment Design for Task-Level Pairs

TL;DR

This paper introduces ATLAS, a method for automatically generating joint curricula of tasks and levels in reinforcement learning, improving training efficiency and policy performance in complex environments.

Contribution

ATLAS is the first approach to co-design tasks and levels simultaneously, advancing unsupervised environment design for complex instruction-following agents.

Findings

ATLAS significantly outperforms random sampling in creating solvable, challenging task-level pairs.

Mutations based on task and level structure accelerate policy convergence.

The evaluation suite effectively measures progress in task-level pair generation.

Abstract

Training general agents to follow complex instructions (tasks) in intricate environments (levels) remains a core challenge in reinforcement learning. Random sampling of task-level pairs often produces unsolvable combinations, highlighting the need to co-design tasks and levels. While unsupervised environment design (UED) has proven effective at automatically designing level curricula, prior work has only considered a fixed task. We present ATLAS (Aligning Tasks and Levels for Autocurricula of Specifications), a novel method that generates joint autocurricula over tasks and levels. Our approach builds upon UED to automatically produce solvable yet challenging task-level pairs for policy training. To evaluate ATLAS and drive progress in the field, we introduce an evaluation suite that models tasks as reward machines in Minigrid levels. Experiments demonstrate that ATLAS vastly outperforms random sampling approaches, particularly when sampling solvable pairs is unlikely. We further show that mutations leveraging the structure of both tasks and levels accelerate convergence to performant policies.