Learning and Understanding a Disentangled Feature Representation for Hidden Parameters in Reinforcement Learning

TL;DR

This paper introduces an unsupervised method using RNN-based world models and metric learning to disentangle and analyze hidden parameters in reinforcement learning environments, aiding understanding of environment dynamics.

Contribution

The paper proposes a novel unsupervised approach that isolates hidden parameter effects in RL by leveraging RNN world models and metric learning to create a meaningful feature space.

Findings

Successfully disentangled hidden parameters across multiple RL environments.

Mapped trajectories into a feature space reflecting differences in hidden parameters.

Provided methods to identify and interpret hidden parameter effects.

Abstract

Hidden parameters are latent variables in reinforcement learning (RL) environments that are constant over the course of a trajectory. Understanding what, if any, hidden parameters affect a particular environment can aid both the development and appropriate usage of RL systems. We present an unsupervised method to map RL trajectories into a feature space where distance represents the relative difference in system behavior due to hidden parameters. Our approach disentangles the effects of hidden parameters by leveraging a recurrent neural network (RNN) world model as used in model-based RL. First, we alter the standard world model training algorithm to isolate the hidden parameter information in the world model memory. Then, we use a metric learning approach to map the RNN memory into a space with a distance metric approximating a bisimulation metric with respect to the hidden parameters. The resulting disentangled feature space can be used to meaningfully relate trajectories to each other and analyze the hidden parameter. We demonstrate our approach on four hidden parameters across three RL environments. Finally we present two methods to help identify and understand the effects of hidden parameters on systems.