Why Do Animals Need Shaping? A Theory of Task Composition and Curriculum Learning

TL;DR

This paper develops a theoretical framework for understanding how curriculum shaping enhances learning of complex, compositional tasks in deep reinforcement learning, revealing the dynamics and benefits of different training strategies.

Contribution

It introduces an analytical model of deep policy gradient learning for compositional tasks, providing insights into shaping strategies and their impact on learning efficiency.

Findings

Pre-training task primitives improves learning speed.

Task complexity influences shaping strategy effectiveness.

Theoretical analysis explains benefits of curriculum shaping in reinforcement learning.

Abstract

Diverse studies in systems neuroscience begin with extended periods of curriculum training known as `shaping' procedures. These involve progressively studying component parts of more complex tasks, and can make the difference between learning a task quickly, slowly or not at all. Despite the importance of shaping to the acquisition of complex tasks, there is as yet no theory that can help guide the design of shaping procedures, or more fundamentally, provide insight into its key role in learning. Modern deep reinforcement learning systems might implicitly learn compositional primitives within their multilayer policy networks. Inspired by these models, we propose and analyse a model of deep policy gradient learning of simple compositional reinforcement learning tasks. Using the tools of statistical physics, we solve for exact learning dynamics and characterise different learning strategies including primitives pre-training, in which task primitives are studied individually before learning compositional tasks. We find a complex interplay between task complexity and the efficacy of shaping strategies. Overall, our theory provides an analytical understanding of the benefits of shaping in a class of compositional tasks and a quantitative account of how training protocols can disclose useful task primitives, ultimately yielding faster and more robust learning.