Maximum diffusion reinforcement learning

TL;DR

This paper introduces maximum diffusion reinforcement learning, a method that decorrelates experiences of embodied agents to enable effective single-shot learning and improve performance in continuous deployment scenarios.

Contribution

It develops a novel approach based on statistical mechanics to decorrelate experiences, extending maximum entropy techniques for reinforcement learning in embodied agents.

Findings

Provably enables single-shot learning in continuous tasks

Generalizes maximum entropy methods

Outperforms state-of-the-art benchmarks

Abstract

Robots and animals both experience the world through their bodies and senses. Their embodiment constrains their experiences, ensuring they unfold continuously in space and time. As a result, the experiences of embodied agents are intrinsically correlated. Correlations create fundamental challenges for machine learning, as most techniques rely on the assumption that data are independent and identically distributed. In reinforcement learning, where data are directly collected from an agent's sequential experiences, violations of this assumption are often unavoidable. Here, we derive a method that overcomes this issue by exploiting the statistical mechanics of ergodic processes, which we term maximum diffusion reinforcement learning. By decorrelating agent experiences, our approach provably enables single-shot learning in continuous deployments over the course of individual task attempts. Moreover, we prove our approach generalizes well-known maximum entropy techniques, and robustly exceeds state-of-the-art performance across popular benchmarks. Our results at the nexus of physics, learning, and control form a foundation for transparent and reliable decision-making in embodied reinforcement learning agents.