Improving Policy Optimization with Generalist-Specialist Learning

TL;DR

This paper introduces a generalist-specialist training framework in deep reinforcement learning that combines broad initial training with targeted specialist fine-tuning, enhancing performance on diverse benchmarks.

Contribution

It proposes a novel training approach that integrates generalist and specialist policies, improving generalization and performance in complex environments.

Findings

The framework improves policy learning on Procgen, Meta-World, and ManiSkill benchmarks.

Specialist training accelerates learning and boosts final performance.

Auxiliary rewards from specialists enhance generalist training.

Abstract

Generalization in deep reinforcement learning over unseen environment variations usually requires policy learning over a large set of diverse training variations. We empirically observe that an agent trained on many variations (a generalist) tends to learn faster at the beginning, yet its performance plateaus at a less optimal level for a long time. In contrast, an agent trained only on a few variations (a specialist) can often achieve high returns under a limited computational budget. To have the best of both worlds, we propose a novel generalist-specialist training framework. Specifically, we first train a generalist on all environment variations; when it fails to improve, we launch a large population of specialists with weights cloned from the generalist, each trained to master a selected small subset of variations. We finally resume the training of the generalist with auxiliary rewards induced by demonstrations of all specialists. In particular, we investigate the timing to start specialist training and compare strategies to learn generalists with assistance from specialists. We show that this framework pushes the envelope of policy learning on several challenging and popular benchmarks including Procgen, Meta-World and ManiSkill.