Structure-Informed Deep Reinforcement Learning for Inventory Management

TL;DR

This paper applies deep reinforcement learning to various inventory management problems, demonstrating competitive performance, interpretability, and the integration of structural insights for practical, data-driven decision-making.

Contribution

It introduces a structure-informed DRL approach that incorporates analytical policy characteristics, enhancing interpretability and robustness in inventory management applications.

Findings

DRL outperforms traditional heuristics across multiple scenarios.

The approach captures known optimal policy structures.

Incorporating analytical insights improves interpretability and robustness.

Abstract

This paper investigates the application of Deep Reinforcement Learning (DRL) to classical inventory management problems, with a focus on practical implementation considerations. We apply a DRL algorithm based on DirectBackprop to several fundamental inventory management scenarios including multi-period systems with lost sales (with and without lead times), perishable inventory management, dual sourcing, and joint inventory procurement and removal. The DRL approach learns policies across products using only historical information that would be available in practice, avoiding unrealistic assumptions about demand distributions or access to distribution parameters. We demonstrate that our generic DRL implementation performs competitively against or outperforms established benchmarks and heuristics across these diverse settings, while requiring minimal parameter tuning. Through examination of the learned policies, we show that the DRL approach naturally captures many known structural properties of optimal policies derived from traditional operations research methods. To further improve policy performance and interpretability, we propose a Structure-Informed Policy Network technique that explicitly incorporates analytically-derived characteristics of optimal policies into the learning process. This approach can help interpretability and add robustness to the policy in out-of-sample performance, as we demonstrate in an example with realistic demand data. Finally, we provide an illustrative application of DRL in a non-stationary setting. Our work bridges the gap between data-driven learning and analytical insights in inventory management while maintaining practical applicability.