Safe Markov Chains for Density Control of ON/OFF Agents with Observed Transitions

TL;DR

This paper introduces a convex optimization framework for controlling the density distribution of agents with ON/OFF modes, incorporating observed actions in the ON mode to enhance decision-making in Markov chain models.

Contribution

It develops a novel control method for Markov chains with observed actions and safety constraints, extending traditional models to include ON/OFF decision policies.

Findings

Convex optimization formulations for density control with safety constraints

Extension from single-action ON mode to multiple actions and stochastic OFF transitions

Demonstration of the model's effectiveness through theoretical analysis

Abstract

This paper presents a convex optimization approach to control the density distribution of autonomous mobile agents with two control modes: ON and OFF. The main new characteristic distinguishing this model from standard Markov decision models is the existence of the ON control mode and its observed actions. When an agent is in the ON mode, it can measure the instantaneous outcome of one of the actions corresponding to the ON mode and decides whether it should take this action or not based on this new observation. If it does not take this action, it switches to the OFF mode where it transitions to the next state based on a predetermined set of transitional probabilities, without making any additional observations. In this decision-making model, each agent acts autonomously according to an ON/OFF decision policy, and the discrete probability distribution for the agent's state evolves according to a discrete-time Markov chain that is a linear function of the stochastic environment (i.e., transition probabilities) and the ON/OFF decision policy. The relevant policy synthesis is formulated as a convex optimization problem where hard safety and convergence constraints are imposed on the resulting Markov matrix. We first consider the case where the ON mode has a single action and the OFF mode has deterministic transitions (rather than stochastic) to demonstrate the model and the ideas behind our approach, which is then generalized to the case where ON mode has multiple actions and OFF mode has stochastic transitions.