Robust Dynamic Material Handling via Adaptive Constrained Evolutionary Reinforcement Learning

TL;DR

This paper introduces ACERL, a novel adaptive constrained evolutionary reinforcement learning method for dynamic material handling, effectively scheduling vehicles in real-time while satisfying constraints and demonstrating robustness across various instances.

Contribution

The paper proposes a new ACERL approach that combines evolutionary strategies with reinforcement learning to improve adaptability and constraint satisfaction in dynamic material handling tasks.

Findings

ACERL outperforms several state-of-the-art algorithms in scheduling tasks.

Policies trained by ACERL satisfy all constraints in diverse scenarios.

ACERL demonstrates robust performance on noisy and unseen instances.

Abstract

Dynamic material handling (DMH) involves the assignment of dynamically arriving material transporting tasks to suitable vehicles in real time for minimising makespan and tardiness. In real-world scenarios, historical task records are usually available, which enables the training of a decision policy on multiple instances consisting of historical records. Recently, reinforcement learning has been applied to solve DMH. Due to the occurrence of dynamic events such as new tasks, adaptability is highly required. Solving DMH is challenging since constraints including task delay should be satisfied. A feedback is received only when all tasks are served, which leads to sparse reward. Besides, making the best use of limited computational resources and historical records for training a robust policy is crucial. The time allocated to different problem instances would highly impact the learning process. To tackle those challenges, this paper proposes a novel adaptive constrained evolutionary reinforcement learning (ACERL) approach, which maintains a population of actors for diverse exploration. ACERL accesses each actor for tackling sparse rewards and constraint violation to restrict the behaviour of the policy. Moreover, ACERL adaptively selects the most beneficial training instances for improving the policy. Extensive experiments on eight training and eight unseen test instances demonstrate the outstanding performance of ACERL compared with several state-of-the-art algorithms. Policies trained by ACERL can schedule the vehicles while fully satisfying the constraints. Additional experiments on 40 unseen noised instances show the robust performance of ACERL. Cross-validation further presents the overall effectiveness of ACREL. Besides, a rigorous ablation study highlights the coordination and benefits of each ingredient of ACERL.