Algebraic optimization of sequential decision problems

TL;DR

This paper explores algebraic methods for optimizing long-term rewards in partially observable Markov decision processes, providing new bounds and computational techniques for solving the resulting quadratic-constrained linear problems.

Contribution

It introduces an algebraic characterization of the feasible set for state aggregation problems and analyzes the critical points of the optimization problem.

Findings

Derived bounds on the number of critical points.

Compared algebraic solutions to traditional optimization methods.

Validated theoretical bounds through experiments.

Abstract

We study the optimization of the expected long-term reward in finite partially observable Markov decision processes over the set of stationary stochastic policies. In the case of deterministic observations, also known as state aggregation, the problem is equivalent to optimizing a linear objective subject to quadratic constraints. We characterize the feasible set of this problem as the intersection of a product of affine varieties of rank one matrices and a polytope. Based on this description, we obtain bounds on the number of critical points of the optimization problem. Finally, we conduct experiments in which we solve the KKT equations or the Lagrange equations over different boundary components of the feasible set, and compare the result to the theoretical bounds and to other constrained optimization methods.