On Separation Between Learning and Control in Partially Observed Markov Decision Processes

TL;DR

This paper proposes a theoretical framework that separates learning and control tasks in partially observed Markov decision processes, enabling the combination of offline model-based control with online learning to improve CPS management.

Contribution

It introduces a novel framework that decouples learning from control in POMDPs, addressing challenges of data volume and model discrepancies in CPS.

Findings

Framework facilitates combining offline control with online learning.

Addresses safety and robustness in CPS control strategies.

Enhances scalability of control in large data environments.

Abstract

Cyber-physical systems (CPS) encounter a large volume of data which is added to the system gradually in real time and not altogether in advance. As the volume of data increases, the domain of the control strategies also increases, and thus it becomes challenging to search for an optimal strategy. Even if an optimal control strategy is found, implementing such strategies with increasing domains is burdensome. To derive an optimal control strategy in CPS, we typically assume an ideal model of the system. Such model-based control approaches cannot effectively facilitate optimal solutions with performance guarantees due to the discrepancy between the model and the actual CPS. Alternatively, traditional supervised learning approaches cannot always facilitate robust solutions using data derived offline. Similarly, applying reinforcement learning approaches directly to the actual CPS might impose significant implications on safety and robust operation of the system. The goal of this chapter is to provide a theoretical framework that aims at separating the control and learning tasks which allows us to combine offline model-based control with online learning approaches, and thus circumvent the challenges in deriving optimal control strategies for CPS.