Approximate Equivariance in Reinforcement Learning

TL;DR

This paper introduces approximately equivariant reinforcement learning algorithms that handle near-symmetries in environments, improving performance and robustness over exact symmetry methods in continuous control and financial tasks.

Contribution

It develops new RL architectures based on relaxed group and steerable convolutions for approximate symmetry, with theoretical analysis and empirical validation.

Findings

Performs comparably to exact equivariant networks with exact symmetries.

Outperforms exact methods in domains with approximate symmetry.

Increases robustness to noise at test time.

Abstract

Equivariant neural networks have shown great success in reinforcement learning, improving sample efficiency and generalization when there is symmetry in the task. However, in many problems, only approximate symmetry is present, which makes imposing exact symmetry inappropriate. Recently, approximately equivariant networks have been proposed for supervised classification and modeling physical systems. In this work, we develop approximately equivariant algorithms in reinforcement learning (RL). We define approximately equivariant MDPs and theoretically characterize the effect of approximate equivariance on the optimal $Q$ function. We propose novel RL architectures using relaxed group and steerable convolutions and experiment on several continuous control domains and stock trading with real financial data. Our results demonstrate that the approximately equivariant network performs on par with exactly equivariant networks when exact symmetries are present, and outperforms them when the domains exhibit approximate symmetry. As an added byproduct of these techniques, we observe increased robustness to noise at test time. Our code is available at https://github.com/jypark0/approx_equiv_rl.