DeepStock: Reinforcement Learning with Policy Regularizations for Inventory Management

TL;DR

This paper introduces policy regularizations based on classical inventory concepts to enhance deep reinforcement learning for inventory management, leading to faster training and better performance, demonstrated through real-world deployment and synthetic experiments.

Contribution

It proposes a novel regularization approach grounded in classical inventory theory to improve DRL methods for inventory management, addressing hyperparameter sensitivity issues.

Findings

Accelerated hyperparameter tuning with policy regularizations

Improved DRL performance in real-world deployment

Synthetic experiments show reshaped effectiveness of DRL methods

Abstract

Deep Reinforcement Learning (DRL) provides a general-purpose methodology for training inventory policies that can leverage big data and compute. However, off-the-shelf implementations of DRL have seen mixed success, often plagued by high sensitivity to the hyperparameters used during training. In this paper, we show that by imposing policy regularizations, grounded in classical inventory concepts such as "Base Stock", we can significantly accelerate hyperparameter tuning and improve the final performance of several DRL methods. We report details from a 100% deployment of DRL with policy regularizations on Alibaba's e-commerce platform, Tmall. We also include extensive synthetic experiments, which show that policy regularizations reshape the narrative on what is the best DRL method for inventory management.