Solving Factored MDPs with Continuous and Discrete Variables

TL;DR

This paper introduces a novel framework for efficiently solving hybrid Markov decision processes with both continuous and discrete variables, leveraging problem structure and a new discretization method.

Contribution

It presents the first structured approach for hybrid MDPs using factored models and a new discretization technique that avoids exponential complexity.

Findings

Effective handling of high-dimensional continuous state spaces.

Improved approximation quality with theoretical bounds.

Successful experiments on complex control problems.

Abstract

Although many real-world stochastic planning problems are more naturally formulated by hybrid models with both discrete and continuous variables, current state-of-the-art methods cannot adequately address these problems. We present the first framework that can exploit problem structure for modeling and solving hybrid problems efficiently. We formulate these problems as hybrid Markov decision processes (MDPs with continuous and discrete state and action variables), which we assume can be represented in a factored way using a hybrid dynamic Bayesian network (hybrid DBN). This formulation also allows us to apply our methods to collaborative multiagent settings. We present a new linear program approximation method that exploits the structure of the hybrid MDP and lets us compute approximate value functions more efficiently. In particular, we describe a new factored discretization of continuous variables that avoids the exponential blow-up of traditional approaches. We provide theoretical bounds on the quality of such an approximation and on its scale-up potential. We support our theoretical arguments with experiments on a set of control problems with up to 28-dimensional continuous state space and 22-dimensional action space.