Computing optimal policies for managing inventories with noisy observations

TL;DR

This paper applies deep reinforcement learning to inventory management with noisy observations, demonstrating how to compute near-optimal policies in partially observable settings and analyzing their structure under Gaussian assumptions.

Contribution

It introduces a DDPG-based approach for partially observable inventory control problems and characterizes the structure of optimal policies in Gaussian cases.

Findings

DDPG effectively computes policies for noisy inventory problems.

Mean beliefs serve as sufficient statistics in Gaussian scenarios.

Numerical results compare DDPG policies with optimal solutions.

Abstract

This paper implements the Deep Deterministic Policy Gradient (DDPG) algorithm for computing optimal policies for partially observable single-product periodic review inventory control problems with setup costs and backorders. The decision maker does not know the exact inventory level, but can obtain noise-corrupted observations of them. The goal is to maximize the expected total discounted costs incurred over a finite planning horizon. We also investigate the Gaussian version of this problem with normally distributed initial inventories, demands, and observation noise. We show that expected posterior observations of inventory levels, also called mean beliefs, provide sufficient statistics for the Gaussian problem. Moreover, they can be represented in the form of a Markov Decision Processes for an inventory control system with time-dependent holding costs and demands. Thus, for a Gaussian problem, the there exist (s_t,S_t)-optimal policies based on mean beliefs, and this fact explains the structure of the approximately optimal policies computed by DDPG. For the Gaussian case, we also numerically compare the performance of policies derived from DDPG to optimal policies for discretized versions of the original continuous problem.