Learning to Optimize Job Shop Scheduling Under Structural Uncertainty

TL;DR

This paper introduces a novel reinforcement learning approach, UP-AAC, for job shop scheduling under structural uncertainty, improving stability and performance by using an asymmetric critic and an attention-based uncertainty perception model.

Contribution

The paper proposes UP-AAC, an innovative actor-critic framework with an asymmetric critic and UPM, specifically addressing structural uncertainty in JSSP, which is not well handled by existing methods.

Findings

Outperforms existing methods in reducing makespan.

Provides more stable learning through lower-variance policy gradients.

Enhances scheduling decisions with an attention-based uncertainty perception model.

Abstract

The Job-Shop Scheduling Problem (JSSP), under various forms of manufacturing uncertainty, has recently attracted considerable research attention. Most existing studies focus on parameter uncertainty, such as variable processing times, and typically adopt the actor-critic framework. In this paper, we explore a different but prevalent form of uncertainty in JSSP: structural uncertainty. Structural uncertainty arises when a job may follow one of several routing paths, and the selection is determined not by policy, but by situational factors (e.g., the quality of intermediate products) that cannot be known in advance. Existing methods struggle to address this challenge due to incorrect credit assignment: a high-quality action may be unfairly penalized if it is followed by a time-consuming path. To address this problem, we propose a novel method named UP-AAC. In contrast to conventional actor-critic methods, UP-AAC employs an asymmetric architecture. While its actor receives a standard stochastic state, the critic is crucially provided with a deterministic state reconstructed in hindsight. This design allows the critic to learn a more accurate value function, which in turn provides a lower-variance policy gradient to the actor, leading to more stable learning. In addition, we design an attention-based Uncertainty Perception Model (UPM) to enhance the actor's scheduling decisions. Extensive experiments demonstrate that our method outperforms existing approaches in reducing makespan on benchmark instances.