Curriculum Reinforcement Learning via Morphology-Environment Co-Evolution

TL;DR

This paper introduces a co-evolution approach for reinforcement learning where both agent morphology and environment are optimized simultaneously, creating an automatic curriculum that enhances generalization to new, unseen environments.

Contribution

The paper proposes a novel morphology-environment co-evolution framework with automatic policy-driven updates, improving RL generalization without manual curriculum design.

Findings

Co-evolution significantly improves generalization in unseen environments.

Automatic environment and morphology updates outperform static or manually designed curricula.

The approach adapts morphology and environment dynamically based on learning progress.

Abstract

Throughout long history, natural species have learned to survive by evolving their physical structures adaptive to the environment changes. In contrast, current reinforcement learning (RL) studies mainly focus on training an agent with a fixed morphology (e.g., skeletal structure and joint attributes) in a fixed environment, which can hardly generalize to changing environments or new tasks. In this paper, we optimize an RL agent and its morphology through ``morphology-environment co-evolution (MECE)'', in which the morphology keeps being updated to adapt to the changing environment, while the environment is modified progressively to bring new challenges and stimulate the improvement of the morphology. This leads to a curriculum to train generalizable RL, whose morphology and policy are optimized for different environments. Instead of hand-crafting the curriculum, we train two policies to automatically change the morphology and the environment. To this end, (1) we develop two novel and effective rewards for the two policies, which are solely based on the learning dynamics of the RL agent; (2) we design a scheduler to automatically determine when to change the environment and the morphology. In experiments on two classes of tasks, the morphology and RL policies trained via MECE exhibit significantly better generalization performance in unseen test environments than SOTA morphology optimization methods. Our ablation studies on the two MECE policies further show that the co-evolution between the morphology and environment is the key to the success.