Meta-reinforcement learning with minimum attention

TL;DR

This paper introduces a minimum attention principle in reinforcement learning, inspired by biological control, leading to faster adaptation, reduced variance, and improved energy efficiency in high-dimensional dynamics.

Contribution

It applies minimum attention as a novel regularization in RL, connecting it to meta-learning and demonstrating empirical advantages over existing algorithms.

Findings

Outperforms state-of-the-art RL algorithms in adaptation speed.

Reduces variance caused by model and environment perturbations.

Enhances energy efficiency in control tasks.

Abstract

Minimum attention applies the least action principle to changes of control concerning state and time, first proposed by Brockett. The involved regularization is highly relevant in emulating biological control, such as motor learning. We apply minimum attention in reinforcement learning (RL) as part of the rewards and investigate its connection to meta-learning and stabilization. Specifically, model-based meta-learning with minimum attention is explored in high-dimensional nonlinear dynamics. Ensemble-based model learning and gradient-based meta-policy learning are alternately performed. Empirically, the minimum attention does show outperforming competence in comparison to the state-of-the-art algorithms of model-free and model-based RL, i.e., fast adaptation in few shots and variance reduction from the perturbations of the model and environment. Furthermore, the minimum attention demonstrates an improvement in energy efficiency.