Centralized Reduction of Decentralized Stochastic Control Models and their weak-Feller Regularity

TL;DR

This paper demonstrates how certain decentralized stochastic control problems can be transformed into centralized MDPs, establishing conditions for regularity and optimal policies, thereby enabling approximation and learning approaches.

Contribution

It extends prior results by providing a unified reduction framework for decentralized control problems with general spaces and analyzing their regularity properties.

Findings

Decentralized problems can be reduced to centralized MDPs under specific information structures.

Conditions for weak-Feller regularity of the transition kernels are established.

Existence of optimal policies and their separated nature are proven.

Abstract

Decentralized stochastic control problems involving general state/measurement/action spaces are intrinsically difficult to study because of the inapplicability of standard tools from centralized (single-agent) stochastic control. In this paper, we address some of these challenges for decentralized stochastic control with standard Borel spaces under two different but tightly related information structures: the one-step delayed information sharing pattern (OSDISP), and the $K$-step periodic information sharing pattern (KSPISP). We will show that the one-step delayed and $K$-step periodic problems can be reduced to a centralized Markov Decision Process (MDP), generalizing prior results which considered finite, linear, or static models, by addressing several measurability and topological questions. We then provide sufficient conditions for the transition kernels of both centralized reductions to be weak-Feller. The existence and separated nature of optimal policies under both information structures are then established. The weak Feller regularity also facilitates rigorous approximation and learning theoretic results, as shown in the paper.