Maximum Total Correlation Reinforcement Learning

TL;DR

This paper introduces a reinforcement learning method that maximizes total correlation in trajectories to promote simplicity, resulting in more robust and efficient policies in simulated robot tasks.

Contribution

It proposes a novel RL algorithm that maximizes total correlation within trajectories, enhancing policy simplicity and robustness beyond existing regularization techniques.

Findings

Policies induce periodic and compressible trajectories

Method improves robustness to noise and dynamic changes

Enhances performance in original tasks

Abstract

Simplicity is a powerful inductive bias. In reinforcement learning, regularization is used for simpler policies, data augmentation for simpler representations, and sparse reward functions for simpler objectives, all that, with the underlying motivation to increase generalizability and robustness by focusing on the essentials. Supplementary to these techniques, we investigate how to promote simple behavior throughout the episode. To that end, we introduce a modification of the reinforcement learning problem that additionally maximizes the total correlation within the induced trajectories. We propose a practical algorithm that optimizes all models, including policy and state representation, based on a lower-bound approximation. In simulated robot environments, our method naturally generates policies that induce periodic and compressible trajectories, and that exhibit superior robustness to noise and changes in dynamics compared to baseline methods, while also improving performance in the original tasks.