Hierarchical Deep Reinforcement Learning Framework for Multi-Year Asset Management Under Budget Constraints

TL;DR

This paper introduces a hierarchical deep reinforcement learning framework for multi-year infrastructure asset management that effectively handles complex decision spaces, budget constraints, and environmental uncertainties, demonstrated through sewer network case studies.

Contribution

It presents a novel hierarchical RL approach with linear programming integration for scalable, budget-compliant multi-year infrastructure planning, outperforming traditional methods.

Findings

Faster convergence compared to baseline algorithms

Effective scalability to larger sewer networks

Consistently near-optimal solutions under budget constraints

Abstract

Budget planning and maintenance optimization are crucial for infrastructure asset management, ensuring cost-effectiveness and sustainability. However, the complexity arising from combinatorial action spaces, diverse asset deterioration, stringent budget constraints, and environmental uncertainty significantly limits existing methods' scalability. This paper proposes a Hierarchical Deep Reinforcement Learning methodology specifically tailored to multi-year infrastructure planning. Our approach decomposes the problem into two hierarchical levels: a high-level Budget Planner allocating annual budgets within explicit feasibility bounds, and a low-level Maintenance Planner prioritizing assets within the allocated budget. By structurally separating macro-budget decisions from asset-level prioritization and integrating linear programming projection within a hierarchical Soft Actor-Critic framework, the method efficiently addresses exponential growth in the action space and ensures rigorous budget compliance. A case study evaluating sewer networks of varying sizes (10, 15, and 20 sewersheds) illustrates the effectiveness of the proposed approach. Compared to conventional Deep Q-Learning and enhanced genetic algorithms, our methodology converges more rapidly, scales effectively, and consistently delivers near-optimal solutions even as network size grows.