On the Optimality Equation for Average Cost Markov Decision Processes and its Validity for Inventory Control

TL;DR

This paper establishes sufficient conditions for the validity of the average-cost optimality equation in Markov Decision Processes with unbounded costs and weakly continuous transitions, applying these results to inventory control problems.

Contribution

It provides new conditions ensuring the validity of the average-cost optimality equation for complex MDPs, including inventory control, and demonstrates the convergence and continuity of value functions.

Findings

Optimality inequality holds as an equality for inventory control.

Convergence of discounted to average-cost value functions is established.

Average-cost optimal policies are derived from the optimality equation.

Abstract

As is well known, average-cost optimality inequalities imply the existence of stationary optimal policies for Markov Decision Processes with average costs per unit time, and these inequalities hold under broad natural conditions. This paper provides sufficient conditions for the validity of the average-cost optimality equation for an infinite state problem with weakly continuous transition probabilities and with possibly unbounded one-step costs and noncompact action sets. These conditions also imply the convergence of sequences of discounted relative value functions to average-cost relative value functions and the continuity of average-cost relative value functions. As shown in the paper, the classic periodic-review inventory control problem satisfies these conditions. Therefore, the optimality inequality holds in the form of an equality with a continuous average-cost relative value function for this problem. In addition, the $K$-convexity of discounted relative value functions and their convergence to average-cost relative value functions, when the discount factor increases to 1, imply the $K$-convexity of average-cost relative value functions. This implies that average-cost optimal $(s,S)$ policies for the inventory control problem can be derived from the average-cost optimality equation.