Reinforcement Learning Using known Invariances

TL;DR

This paper introduces a framework for incorporating known symmetries into kernel-based reinforcement learning, leading to improved sample efficiency and performance.

Contribution

It develops a symmetry-aware variant of optimistic least-squares value iteration that leverages invariant kernels and provides theoretical analysis of sample efficiency gains.

Findings

Symmetry-aware RL outperforms standard kernel methods in experiments.

Theoretical bounds on information gain and covering numbers are established.

Empirical results show significant performance improvements.

Abstract

In many real-world reinforcement learning (RL) problems, the environment exhibits inherent symmetries that can be exploited to improve learning efficiency. This paper develops a theoretical and algorithmic framework for incorporating known group symmetries into kernel-based RL. We propose a symmetry-aware variant of optimistic least-squares value iteration (LSVI), which leverages invariant kernels to encode invariance in both rewards and transition dynamics. Our analysis establishes new bounds on the maximum information gain and covering numbers for invariant RKHSs, explicitly quantifying the sample efficiency gains from symmetry. Empirical results on a customized Frozen Lake environment and a 2D placement design problem confirm the theoretical improvements, demonstrating that symmetry-aware RL achieves significantly better performance than their standard kernel counterparts. These findings highlight the value of structural priors in designing more sample-efficient reinforcement learning algorithms.