Towards Learning Controllable Representations of Physical Systems

TL;DR

This paper proposes metrics to evaluate learned representations of dynamical systems for control, focusing on their correspondence to true states, and demonstrates their predictive power for RL performance in simulated tasks.

Contribution

It introduces principled metrics based on true state correspondence to evaluate representations for control, moving beyond downstream RL performance assessment.

Findings

Metrics predict RL performance in simulated tasks

High mutual information correlates with better control

Temporal smoothness aligns with effective representations

Abstract

Learned representations of dynamical systems reduce dimensionality, potentially supporting downstream reinforcement learning (RL). However, no established methods predict a representation's suitability for control and evaluation is largely done via downstream RL performance, slowing representation design. Towards a principled evaluation of representations for control, we consider the relationship between the true state and the corresponding representations, proposing that ideally each representation corresponds to a unique true state. This motivates two metrics: temporal smoothness and high mutual information between true state/representation. These metrics are related to established representation objectives, and studied on Lagrangian systems where true state, information requirements, and statistical properties of the state can be formalized for a broad class of systems. These metrics are shown to predict reinforcement learning performance in a simulated peg-in-hole task when comparing variants of autoencoder-based representations.