Stochastic dynamic programming with non-linear discounting

TL;DR

This paper investigates stochastic dynamic programming with non-linear discount functions in Markov decision processes, proving the existence of solutions to Bellman equations and optimal stationary policies under certain boundedness conditions.

Contribution

It extends existing models by establishing solution existence and optimal policies for non-linear discounting in Markov decision processes with Borel state spaces.

Findings

Bellman equation has a solution under specified conditions.

Existence of optimal stationary policy in infinite horizon.

Applicable to bounded and unbounded utility cases.

Abstract

In this paper, we study a Markov decision process with a non-linear discount function and with a Borel state space. We define a recursive discounted utility, which resembles non-additive utility functions considered in a number of models in economics. Non-additivity here follows from non-linearity of the discount function. Our study is complementary to the work of Ja\'skiewicz, Matkowski and Nowak (Math. Oper. Res. 38 (2013), 108-121), where also non-linear discounting is used in the stochastic setting, but the expectation of utilities aggregated on the space of all histories of the process is applied leading to a non-stationary dynamic programming model. Our aim is to prove that in the recursive discounted utility case the Bellman equation has a solution and there exists an optimal stationary policy for the problem in the infinite time horizon. Our approach includes two cases: $(a)$ when the one-stage utility is bounded on both sides by a weight function multiplied by some positive and negative constants, and $(b)$ when the one-stage utility is unbounded from below.