Locally Constrained Representations in Reinforcement Learning

TL;DR

This paper introduces locally constrained representations in reinforcement learning, using an auxiliary loss to improve the stability and robustness of learned state representations, leading to better performance especially in continuous control tasks.

Contribution

It proposes a novel auxiliary loss that constrains state representations to be predictable from neighboring states, enhancing representation stability in RL.

Findings

Significant performance improvements in continuous control benchmarks.

Enhanced stability and robustness of learned representations.

Better disentanglement of task-specific features from environment dynamics.

Abstract

The success of Reinforcement Learning (RL) heavily relies on the ability to learn robust representations from the observations of the environment. In most cases, the representations learned purely by the reinforcement learning loss can differ vastly across states depending on how the value functions change. However, the representations learned need not be very specific to the task at hand. Relying only on the RL objective may yield representations that vary greatly across successive time steps. In addition, since the RL loss has a changing target, the representations learned would depend on how good the current values/policies are. Thus, disentangling the representations from the main task would allow them to focus not only on the task-specific features but also the environment dynamics. To this end, we propose locally constrained representations, where an auxiliary loss forces the state representations to be predictable by the representations of the neighboring states. This encourages the representations to be driven not only by the value/policy learning but also by an additional loss that constrains the representations from over-fitting to the value loss. We evaluate the proposed method on several known benchmarks and observe strong performance. Especially in continuous control tasks, our experiments show a significant performance improvement.