Enhancing the Hierarchical Environment Design via Generative Trajectory Modeling

TL;DR

This paper proposes a hierarchical environment design framework using generative trajectory modeling to efficiently create training environments, improving resource utilization and agent generalization in Unsupervised Environment Design.

Contribution

It introduces a hierarchical MDP framework with a generative modeling approach for environment design under resource constraints, enabling more effective curriculum generation.

Findings

Reduces resource consumption in environment generation.

Improves agent zero-shot transfer performance.

Demonstrates effectiveness across multiple domains.

Abstract

Unsupervised Environment Design (UED) is a paradigm for automatically generating a curriculum of training environments, enabling agents trained in these environments to develop general capabilities, i.e., achieving good zero-shot transfer performance. However, existing UED approaches focus primarily on the random generation of environments for open-ended agent training. This is impractical in scenarios with limited resources, such as the constraints on the number of generated environments. In this paper, we introduce a hierarchical MDP framework for environment design under resource constraints. It consists of an upper-level RL teacher agent that generates suitable training environments for a lower-level student agent. The RL teacher can leverage previously discovered environment structures and generate environments at the frontier of the student's capabilities by observing the student policy's representation. Moreover, to reduce the time-consuming collection of experiences for the upper-level teacher, we utilize recent advances in generative modeling to synthesize a trajectory dataset to train the teacher agent. Our proposed method significantly reduces the resource-intensive interactions between agents and environments and empirical experiments across various domains demonstrate the effectiveness of our approach.