Efficient Unsupervised Environment Design through Hierarchical Policy Representation Learning

TL;DR

This paper proposes a hierarchical MDP framework for efficient unsupervised environment design, enabling the creation of training environments based on student capabilities with fewer interactions, suitable for resource-limited scenarios.

Contribution

It introduces a hierarchical MDP approach with a generative model to improve environment design efficiency and reduce interaction requirements.

Findings

Outperforms baseline methods in multiple domains

Requires fewer teacher-student interactions

Effective in resource-constrained settings

Abstract

Unsupervised Environment Design (UED) has emerged as a promising approach to developing general-purpose agents through automated curriculum generation. Popular UED methods focus on Open-Endedness, where teacher algorithms rely on stochastic processes for infinite generation of useful environments. This assumption becomes impractical in resource-constrained scenarios where teacher-student interaction opportunities are limited. To address this challenge, we introduce a hierarchical Markov Decision Process (MDP) framework for environment design. Our framework features a teacher agent that leverages student policy representations derived from discovered evaluation environments, enabling it to generate training environments based on the student's capabilities. To improve efficiency, we incorporate a generative model that augments the teacher's training dataset with synthetic data, reducing the need for teacher-student interactions. In experiments across several domains, we show that our method outperforms baseline approaches while requiring fewer teacher-student interactions in a single episode. The results suggest the applicability of our approach in settings where training opportunities are limited.