Continuous MDP Homomorphisms and Homomorphic Policy Gradient

TL;DR

This paper extends MDP homomorphisms to continuous control environments, deriving a policy gradient theorem and proposing an actor-critic algorithm that leverages environment symmetries for improved reinforcement learning from pixel data.

Contribution

It introduces a novel continuous MDP homomorphism framework and a simultaneous learning algorithm, enhancing RL efficiency and generalization in continuous and pixel-based settings.

Findings

Improved performance on DeepMind Control Suite benchmarks.

Effective learning of policies and homomorphisms from pixel observations.

Utilization of environment symmetries enhances policy optimization.

Abstract

Abstraction has been widely studied as a way to improve the efficiency and generalization of reinforcement learning algorithms. In this paper, we study abstraction in the continuous-control setting. We extend the definition of MDP homomorphisms to encompass continuous actions in continuous state spaces. We derive a policy gradient theorem on the abstract MDP, which allows us to leverage approximate symmetries of the environment for policy optimization. Based on this theorem, we propose an actor-critic algorithm that is able to learn the policy and the MDP homomorphism map simultaneously, using the lax bisimulation metric. We demonstrate the effectiveness of our method on benchmark tasks in the DeepMind Control Suite. Our method's ability to utilize MDP homomorphisms for representation learning leads to improved performance when learning from pixel observations.