Self-Triggered Markov Decision Processes

TL;DR

This paper extends self-triggered control strategies to general Markov Decision Processes, proposing novel policies that optimize resource use and system performance through dynamic programming and theoretical analysis.

Contribution

It introduces a new framework for self-triggered policies in MDPs, including a DP equation with lookahead and methods for balancing resource consumption and performance.

Findings

Effective reduction in communication resources demonstrated.

Trade-offs between resource use and system performance analyzed.

Theoretical foundations for policy computation established.

Abstract

In this paper, we study Markov Decision Processes (MDPs) with self-triggered strategies, where the idea of self-triggered control is extended to more generic MDP models. This extension broadens the application of self-triggering policies to a broader range of systems. We study the co-design problems of the control policy and the triggering policy to optimize two pre-specified cost criteria. The first cost criterion is introduced by incorporating a pre-specified update penalty into the traditional MDP cost criteria to reduce the use of communication resources. Under this criteria, a novel dynamic programming (DP) equation called DP equation with optimized lookahead to proposed to solve for the self-triggering policy under this criteria. The second self-triggering policy is to maximize the triggering time while still guaranteeing a pre-specified level of sub-optimality. Theoretical underpinnings are established for the computation and implementation of both policies. Through a gridworld numerical example, we illustrate the two policies' effectiveness in reducing sources consumption and demonstrate the trade-offs between resource consumption and system performance.