Reinforcement Learning for Jointly Optimal Coding and Control Policies for a Controlled Markovian System over a Communication Channel

TL;DR

This paper develops theoretical foundations and reinforcement learning methods for jointly optimizing coding and control policies in controlled Markov systems over finite-rate communication channels, achieving near optimality with finite models.

Contribution

It introduces regularity, existence, and structural results for optimal policies, and demonstrates near optimality and RL convergence of finite model approximations in joint coding-control problems.

Findings

Finite model approximations are near optimal.

Reinforcement learning converges to near optimal policies.

Comparison of approximation schemes and RL performance.

Abstract

We study the problem of joint optimization involving coding and control policies for a controlled Markovian sytem over a finite-rate noiseless communication channel. While structural results on the optimal encoding and control have been obtained in the literature, their implementation has been prohibitive in general, except for linear models. We develop regularity and existence results on optimal policies. We then obtain rigorous approximation and near optimality results for jointly optimal coding and control policies. To this end, we first develop existence, regularity, and structural properties on optimal policies, followed by rigorous approximations and reinforcement learning results. Notably, we establish near optimality of finite model approximations obtained via predictor quantization as well as sliding finite window approximations, and their reinforcement learning convergence to near optimality. A detailed comparison of the approximation schemes and their reinforcement learning performance is presented.