Procedural Content Generation: Better Benchmarks for Transfer Reinforcement Learning

TL;DR

This paper reviews transfer reinforcement learning, highlighting the late adoption of deep learning, ongoing challenges in generalization, and the need for unified benchmarks and procedural content generation to advance the field.

Contribution

It provides a comprehensive analysis of current TRL research, identifies key challenges, and suggests future directions including benchmarks and procedural content generation.

Findings

Deep learning adoption in TRL started in 2018.

Generalization remains a major challenge in TRL.

Emerging benchmarks like Alchemy and Meta-World are promising.

Abstract

The idea of transfer in reinforcement learning (TRL) is intriguing: being able to transfer knowledge from one problem to another problem without learning everything from scratch. This promises quicker learning and learning more complex methods. To gain an insight into the field and to detect emerging trends, we performed a database search. We note a surprisingly late adoption of deep learning that starts in 2018. The introduction of deep learning has not yet solved the greatest challenge of TRL: generalization. Transfer between different domains works well when domains have strong similarities (e.g. MountainCar to Cartpole), and most TRL publications focus on different tasks within the same domain that have few differences. Most TRL applications we encountered compare their improvements against self-defined baselines, and the field is still missing unified benchmarks. We consider this to be a disappointing situation. For the future, we note that: (1) A clear measure of task similarity is needed. (2) Generalization needs to improve. Promising approaches merge deep learning with planning via MCTS or introduce memory through LSTMs. (3) The lack of benchmarking tools will be remedied to enable meaningful comparison and measure progress. Already Alchemy and Meta-World are emerging as interesting benchmark suites. We note that another development, the increase in procedural content generation (PCG), can improve both benchmarking and generalization in TRL.