What deep reinforcement learning tells us about human motor learning and vice-versa

TL;DR

This paper compares deep reinforcement learning algorithms to human motor learning, finds current algorithms lacking, and introduces a new method, MB-DPG, that better mimics human error-based learning and improves learning speed and robustness.

Contribution

The paper introduces MB-DPG, a novel deep RL algorithm inspired by error-based learning, bridging the gap between AI models and human motor adaptation.

Findings

Existing deep RL algorithms do not replicate human motor learning behaviors.

MB-DPG captures human error-based learning in mirror and rotational perturbations.

MB-DPG learns faster and is more robust than traditional RL algorithms on complex tasks.

Abstract

Machine learning and specifically reinforcement learning (RL) has been extremely successful in helping us to understand neural decision making processes. However, RL's role in understanding other neural processes especially motor learning is much less well explored. To explore this connection, we investigated how recent deep RL methods correspond to the dominant motor learning framework in neuroscience, error-based learning. Error-based learning can be probed using a mirror reversal adaptation paradigm, where it produces distinctive qualitative predictions that are observed in humans. We therefore tested three major families of modern deep RL algorithm on a mirror reversal perturbation. Surprisingly, all of the algorithms failed to mimic human behaviour and indeed displayed qualitatively different behaviour from that predicted by error-based learning. To fill this gap, we introduce a novel deep RL algorithm: model-based deterministic policy gradients (MB-DPG). MB-DPG draws inspiration from error-based learning by explicitly relying on the observed outcome of actions. We show MB-DPG captures (human) error-based learning under mirror-reversal and rotational perturbation. Next, we demonstrate error-based learning in the form of MB-DPG learns faster than canonical model-free algorithms on complex arm-based reaching tasks, while being more robust to (forward) model misspecification than model-based RL. These findings highlight the gap between current deep RL methods and human motor adaptation and offer a route to closing this gap, facilitating future beneficial interaction between between the two fields.