Symmetry Detection in Trajectory Data for More Meaningful Reinforcement Learning Representations

TL;DR

This paper introduces a method to automatically detect symmetries in reinforcement learning systems from raw trajectory data, enabling the creation of more meaningful and invariant representations of the state space.

Contribution

The authors propose a novel approach using RNN discriminators to identify symmetries directly from trajectory data without active system control, applicable to various RL environments.

Findings

Successfully identified environment and policy symmetries in simulated RL tasks

Enabled creation of invariant high-level state representations

Demonstrated effectiveness on robotic and UAV control scenarios

Abstract

Knowledge of the symmetries of reinforcement learning (RL) systems can be used to create compressed and semantically meaningful representations of a low-level state space. We present a method of automatically detecting RL symmetries directly from raw trajectory data without requiring active control of the system. Our method generates candidate symmetries and trains a recurrent neural network (RNN) to discriminate between the original trajectories and the transformed trajectories for each candidate symmetry. The RNN discriminator's accuracy for each candidate reveals how symmetric the system is under that transformation. This information can be used to create high-level representations that are invariant to all symmetries on a dataset level and to communicate properties of the RL behavior to users. We show in experiments on two simulated RL use cases (a pusher robot and a UAV flying in wind) that our method can determine the symmetries underlying both the environment physics and the trained RL policy.