Hierarchical Decentralized Stochastic Control for Cyber-Physical Systems

TL;DR

This paper develops a hierarchical decentralized control framework for cyber-physical systems, analyzing two optimization approaches with different reward horizons, and provides theoretical guarantees and bounds for their policies.

Contribution

It introduces two novel optimization frameworks, COpt and FOpt, for hierarchical decentralized control in CPS, with comprehensive analysis and convergence proofs.

Findings

COpt always outperforms FOpt in value function.

Existence of optimal policies is proven for both frameworks.

Explicit bounds on the difference between COpt and FOpt are derived.

Abstract

This paper introduces a two-timescale hierarchical decentralized control architecture for Cyber-Physical Systems (CPS). The system consists of a global controller (GC), and N local controllers (LCs). The GC operates at a slower timescale, imposing budget constraints on the actions of LCs, which function at a faster timescale. Applications can be found in energy grid planning, wildfire management, and other decentralized resource allocation problems. We propose and analyze two optimization frameworks for this setting: COpt and FOpt. In COpt, both GC and LCs together optimize infinite-horizon discounted rewards, while in FOpt the LCs optimize finite-horizon episodic rewards, and the GC optimizes infinite-horizon rewards. Although both frameworks share identical reward functions, their differing horizons can lead to different optimal policies. In particular, FOpt grants greater autonomy to LCs by allowing their policies to be determined only by local objectives, unlike COpt. To our knowledge, these frameworks have not been studied in the literature. We establish the formulations, prove the existence of optimal policies, and prove the convergence of their value iteration algorithms. We further show that COpt always achieves a higher value function than FOpt and derive explicit bounds on their difference. Finally, we establish a set of sufficient structural conditions under which the two frameworks become equivalent.