# Positivity-hardness results on Markov decision processes

## TL;DR

This paper proves that many optimization problems in Markov decision processes are as hard as the longstanding open Positivity problem in number theory, indicating no efficient solutions are likely without major breakthroughs.

## Contribution

It establishes Positivity-hardness for various MDP optimization problems, linking their decidability to a major open problem in number theory.

## Key findings

- Optimization problems are Positivity-hard for MDPs.
- Decidability of these problems is linked to the open Positivity problem.
- No efficient algorithms are likely without breakthroughs in number theory.

## Abstract

This paper investigates a series of optimization problems for one-counter Markov decision processes (MDPs) and integer-weighted MDPs with finite state space. Specifically, it considers problems addressing termination probabilities and expected termination times for one-counter MDPs, as well as satisfaction probabilities of energy objectives, conditional and partial expectations, satisfaction probabilities of constraints on the total accumulated weight, the computation of quantiles for the accumulated weight, and the conditional value-at-risk for accumulated weights for integer-weighted MDPs. Although algorithmic results are available for some special instances, the decidability status of the decision versions of these problems is unknown in general. The paper demonstrates that these optimization problems are inherently mathematically difficult by providing polynomial-time reductions from the Positivity problem for linear recurrence sequences. This problem is a well-known number-theoretic problem whose decidability status has been open for decades and it is known that decidability of the Positivity problem would have far-reaching consequences in analytic number theory. So, the reductions presented in the paper show that an algorithmic solution to any of the investigated problems is not possible without a major breakthrough in analytic number theory. The reductions rely on the construction of MDP-gadgets that encode the initial values and linear recurrence relations of linear recurrence sequences. These gadgets can flexibly be adjusted to prove the various Positivity-hardness results.