Continuous MDP Homomorphisms and Homomorphic Policy Gradient
Sahand Rezaei-Shoshtari, Rosie Zhao, Prakash Panangaden, David Meger,, Doina Precup

TL;DR
This paper extends MDP homomorphisms to continuous control environments, deriving a policy gradient theorem and proposing an actor-critic algorithm that leverages environment symmetries for improved reinforcement learning from pixel data.
Contribution
It introduces a novel continuous MDP homomorphism framework and a simultaneous learning algorithm, enhancing RL efficiency and generalization in continuous and pixel-based settings.
Findings
Improved performance on DeepMind Control Suite benchmarks.
Effective learning of policies and homomorphisms from pixel observations.
Utilization of environment symmetries enhances policy optimization.
Abstract
Abstraction has been widely studied as a way to improve the efficiency and generalization of reinforcement learning algorithms. In this paper, we study abstraction in the continuous-control setting. We extend the definition of MDP homomorphisms to encompass continuous actions in continuous state spaces. We derive a policy gradient theorem on the abstract MDP, which allows us to leverage approximate symmetries of the environment for policy optimization. Based on this theorem, we propose an actor-critic algorithm that is able to learn the policy and the MDP homomorphism map simultaneously, using the lax bisimulation metric. We demonstrate the effectiveness of our method on benchmark tasks in the DeepMind Control Suite. Our method's ability to utilize MDP homomorphisms for representation learning leads to improved performance when learning from pixel observations.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
Taxonomy
TopicsReinforcement Learning in Robotics · Receptor Mechanisms and Signaling · Adversarial Robustness in Machine Learning
