Dynamic Repair and Maintenance of Heterogeneous Machines Dispersed on a Network: A Rollout Method for Online Reinforcement Learning

TL;DR

This paper introduces a novel online reinforcement learning approach for the dynamic maintenance of heterogeneous machines on a network, utilizing a rollout-based policy improvement method that adapts in real-time to system changes.

Contribution

It develops a new online policy improvement technique using rollout methods and an index heuristic for networked machine maintenance, with proven optimality in special cases.

Findings

The OPI heuristic achieves near-optimal performance in fast-changing systems.

The approach is suitable for real-time online implementation with high-frequency state transitions.

Extensive experiments validate the effectiveness of the proposed method.

Abstract

We consider a problem in which a single repairer is responsible for the maintenance and repair of a collection of machines, positioned at different locations on a network of nodes and edges. Machines deteriorate according to stochastic processes and incur increasing costs as they approach complete failure. The times needed for repairs to be performed, and the amounts of time needed for the repairer to switch between different machines, are random and machine-dependent. The problem is formulated as a Markov decision process (MDP) in which the objective is to minimize long-run average costs. We prove the equivalence of an alternative formulation based on rewards and use this to develop an index heuristic policy, which is shown to be optimal in certain special cases. We then use rollout-based reinforcement learning techniques to develop a novel online policy improvement (OPI) approach, which uses the index heuristic as a base policy and also as an insurance option at decision epochs where the best action cannot be selected with sufficient confidence. Results from extensive numerical experiments, involving randomly-generated network layouts and parameter values, show that the OPI heuristic is able to achieve close-to-optimal performance in fast-changing systems with state transitions occurring 100 times per second, suggesting that it is suitable for online implementation.