Absorbing Markov Decision Processes: Geometric Properties and Sufficiency of Finite Mixtures of Deterministic Policies

TL;DR

This paper explores the geometric structure of occupancy measures in Markov decision processes and proves that finite mixtures of deterministic policies are sufficient for representing optimal policies in absorbing MDPs.

Contribution

It characterizes the geometric properties of occupancy measures and establishes the sufficiency of finite mixtures of deterministic policies in absorbing MDPs.

Findings

Geometric properties of occupancy measures are fully characterized.

Finite mixtures of deterministic policies are sufficient for optimality.

Minimal mixture order for policy representation is determined.

Abstract

In this paper we investigate several geometric properties of the set of occupancy measures. In particular, we analyse the structure of the faces generated by a given occupancy measure, together with their relative algebraic interior. We also determine the affine hulls of these faces and describe the associated parallel linear subspaces. It is shown that these structures can be fully characterised in terms of the parameters that define the underlying Markov decision process (MDP). Moreover, we establish that the class of finite mixtures of deterministic stationary policies constitutes a sufficient class of policies for uniformly absorbing MDPs with a measurable state space and multiple criteria. We also provide a characterisation of the minimal order required for a finite mixture of deterministic stationary policies to represent the performance vector of an arbitrary policy.