Solving Factored MDPs with Hybrid State and Action Variables

TL;DR

This paper introduces a hybrid factored MDP model and a corresponding approximate linear programming framework, enabling efficient solutions for large decision problems with mixed variables, demonstrated through theoretical analysis and practical experiments.

Contribution

The paper presents a novel hybrid factored MDP model and the HALP framework, advancing the ability to efficiently solve large-scale hybrid decision problems.

Findings

HALP effectively approximates the optimal value function.

The approach scales to large hybrid decision problems.

Theoretical analysis supports the method's efficiency.

Abstract

Efficient representations and solutions for large decision problems with continuous and discrete variables are among the most important challenges faced by the designers of automated decision support systems. In this paper, we describe a novel hybrid factored Markov decision process (MDP) model that allows for a compact representation of these problems, and a new hybrid approximate linear programming (HALP) framework that permits their efficient solutions. The central idea of HALP is to approximate the optimal value function by a linear combination of basis functions and optimize its weights by linear programming. We analyze both theoretical and computational aspects of this approach, and demonstrate its scale-up potential on several hybrid optimization problems.