End-Effect Exploration Drive for Effective Motor Learning

TL;DR

This paper introduces end-effect drives as a lightweight, goal-directed motor learning method that uses simple effect models and variational free energy minimization, leading to faster training in reinforcement learning.

Contribution

It proposes a novel end-effect drive approach that replaces complex forward models with statistical effect recordings for efficient, goal-oriented motor learning.

Findings

Faster training compared to traditional off-policy methods.

Effective exploration policy derived from uniform target distributions.

Combines intrinsic curiosity with extrinsic rewards for improved learning.

Abstract

Stemming on the idea that a key objective in reinforcement learning is to invert a target distribution of effects, end-effect drives are proposed as an effective way to implement goal-directed motor learning, in the absence of an explicit forward model. An end-effect model relies on a simple statistical recording of the effect of the current policy, here used as a substitute for the more resource-demanding forward models. When combined with a reward structure, it forms the core of a lightweight variational free energy minimization setup. The main difficulty lies in the maintenance of this simplified effect model together with the online update of the policy. When the prior target distribution is uniform, it provides a ways to learn an efficient exploration policy, consistently with the intrinsic curiosity principles. When combined with an extrinsic reward, our approach is finally shown to provide a faster training than traditional off-policy techniques.