Multi-echelon Supply Chains with Uncertain Seasonal Demands and Lead Times Using Deep Reinforcement Learning

TL;DR

This paper applies Deep Reinforcement Learning, specifically PPO2, to optimize production and distribution in multi-echelon supply chains with uncertain demands and lead times, demonstrating its effectiveness over traditional methods.

Contribution

It introduces a Deep RL approach for complex supply chain problems with stochastic demands and lead times, showing improved performance over baseline models.

Findings

PPO2 outperforms baseline in stochastic lead time scenarios by 7.3-11.2%.

PPO2 is more effective with uncertain, non-seasonal demands in constant lead time scenarios.

Greater demand and lead time uncertainty increase the effectiveness of Deep RL methods.

Abstract

We address the problem of production planning and distribution in multi-echelon supply chains. We consider uncertain demands and lead times which makes the problem stochastic and non-linear. A Markov Decision Process formulation and a Non-linear Programming model are presented. As a sequential decision-making problem, Deep Reinforcement Learning (RL) is a possible solution approach. This type of technique has gained a lot of attention from Artificial Intelligence and Optimization communities in recent years. Considering the good results obtained with Deep RL approaches in different areas there is a growing interest in applying them in problems from the Operations Research field. We have used a Deep RL technique, namely Proximal Policy Optimization (PPO2), to solve the problem considering uncertain, regular and seasonal demands and constant or stochastic lead times. Experiments are carried out in different scenarios to better assess the suitability of the algorithm. An agent based on a linearized model is used as a baseline. Experimental results indicate that PPO2 is a competitive and adequate tool for this type of problem. PPO2 agent is better than baseline in all scenarios with stochastic lead times (7.3-11.2%), regardless of whether demands are seasonal or not. In scenarios with constant lead times, the PPO2 agent is better when uncertain demands are non-seasonal (2.2-4.7%). The results show that the greater the uncertainty of the scenario, the greater the viability of this type of approach.