Compactly Restrictable Metric Policy Optimization Problems

TL;DR

This paper introduces a class of metric policy optimization problems called CR-MPOPs, which are designed to be both expressive for robotic systems and solvable via dynamic programming, with theoretical insights into their properties.

Contribution

The paper defines CR-MPOPs, a new class of optimization problems for deterministic MDPs with metric spaces, and establishes their well-posedness and relevance to control systems.

Findings

CR-MPOPs can be characterized using forward-invariance.

CR-MPOPs admit solutions via value iteration.

Theoretical results relate CR-MPOPs to feedback linearizable systems.

Abstract

We study policy optimization problems for deterministic Markov decision processes (MDPs) with metric state and action spaces, which we refer to as Metric Policy Optimization Problems (MPOPs). Our goal is to establish theoretical results on the well-posedness of MPOPs that can characterize practically relevant continuous control systems. To do so, we define a special class of MPOPs called Compactly Restrictable MPOPs (CR-MPOPs), which are flexible enough to capture the complex behavior of robotic systems but specific enough to admit solutions using dynamic programming methods such as value iteration. We show how to arrive at CR-MPOPs using forward-invariance. We further show that our theoretical results on CR-MPOPs can be used to characterize feedback linearizable control affine systems.