A Universal Dynamic Program and Refined Existence Results for Decentralized Stochastic Control

TL;DR

This paper introduces a universal dynamic programming framework for decentralized stochastic control, establishing well-posedness and new existence results for optimal policies, applicable to static and dynamic teams with independent measurements.

Contribution

It develops a general dynamic programming formulation for decentralized control problems, extending previous results and providing new existence theorems under mild conditions.

Findings

A universal dynamic programming recursion is established.

Existence of optimal policies is proven under continuity and compactness conditions.

Counterexamples show weaker conditions may fail to guarantee optimal policies.

Abstract

For sequential stochastic control problems with standard Borel measurement and control action spaces, we introduce a general (universally applicable) dynamic programming formulation, establish its well-posedness, and provide new existence results for optimal policies. Our dynamic program builds in part on Witsenhausen's standard form, but with a different formulation for the state, action, and transition dynamics. Using recent results on measurability properties of strategic measures in decentralized control, we obtain a standard Borel controlled Markov model. This allows for a well-defined dynamic programming recursion through universal measurability properties of the value functions for each time stage. In addition, new existence results are obtained for optimal policies in decentralized stochastic control. These state that for a static team with independent measurements, it suffices for the cost function to be continuous (only) in the actions for the existence of an optimal policy under mild compactness (or tightness) conditions. These also apply to dynamic teams which admit static reductions with independent measurements through a change of measure transformation. We show through a counterexample that weaker conditions may not lead to existence of an optimal team policy. The paper's existence results generalize those previously reported in the literature. A summary of and comparison with previously reported results and some applications are presented.