An exact solution in Markov decision process with multiplicative rewards as a general framework

TL;DR

This paper introduces a novel framework for Markov decision processes with multiplicative rewards, providing exact solutions that depend on noise and generalize traditional additive reward models.

Contribution

It develops an exact, noise-dependent solution for MDPs with multiplicative rewards, extending the classical linear quadratic regulation framework.

Findings

Exact solution for MDPs with multiplicative rewards

Optimal policy depends on noise, unlike traditional models

Framework generalizes additive reward models with arbitrary precision

Abstract

We develop an exactly solvable framework of Markov decision process with a finite horizon, and continuous state and action spaces. We first review the exact solution of conventional linear quadratic regulation with a linear transition and a Gaussian noise, whose optimal policy does not depend on the Gaussian noise, which is an undesired feature in the presence of significant noises. It motivates us to investigate exact solutions which depend on noise. To do so, we generalize the reward accumulation to be a general binary commutative and associative operation. By a new multiplicative accumulation, we obtain an exact solution of optimization assuming linear transitions with a Gaussian noise and the optimal policy is noise dependent in contrast to the additive accumulation. Furthermore, we also show that the multiplicative scheme is a general framework that covers the additive one with an arbitrary precision, which is a model-independent principle.