Managing engineering systems with large state and action spaces through deep reinforcement learning

TL;DR

This paper introduces a deep reinforcement learning framework, DCMAC, for managing large-scale engineering systems with high-dimensional states and actions, enabling efficient decision-making in complex multi-component environments.

Contribution

The paper presents DCMAC, a novel off-policy actor-critic DRL approach with a factorized action representation for large multi-component systems, improving scalability and policy quality.

Findings

DCMAC outperforms Deep Q-Network (DQN) solutions.

DCMAC surpasses optimized baseline policies.

Effective handling of high-dimensional state and action spaces.

Abstract

Decision-making for engineering systems can be efficiently formulated as a Markov Decision Process (MDP) or a Partially Observable MDP (POMDP). Typical MDP and POMDP solution procedures utilize offline knowledge about the environment and provide detailed policies for relatively small systems with tractable state and action spaces. However, in large multi-component systems the sizes of these spaces easily explode, as system states and actions scale exponentially with the number of components, whereas environment dynamics are difficult to be described in explicit forms for the entire system and may only be accessible through numerical simulators. In this work, to address these issues, an integrated Deep Reinforcement Learning (DRL) framework is introduced. The Deep Centralized Multi-agent Actor Critic (DCMAC) is developed, an off-policy actor-critic DRL approach, providing efficient life-cycle policies for large multi-component systems operating in high-dimensional spaces. Apart from deep function approximations that parametrize large state spaces, DCMAC also adopts a factorized representation of the system actions, being able to designate individualized component- and subsystem-level decisions, while maintaining a centralized value function for the entire system. DCMAC compares well against Deep Q-Network (DQN) solutions and exact policies, where applicable, and outperforms optimized baselines that are based on time-based, condition-based and periodic policies.