Time-Varying Parameters in Sequential Decision Making Problems

TL;DR

This paper introduces a control-theoretic framework for optimizing time-varying parameters and decision policies in sequential decision making problems, ensuring cost minimization and parameter tracking.

Contribution

It develops a novel control-based approach using a Maximum Entropy Principle to simultaneously optimize parameters and decision policies in SDMs with time-varying dynamics.

Findings

Parameters asymptotically track local optima

Control law is Lipschitz continuous and bounded

Method effectively minimizes cumulative cost

Abstract

In this paper we address the class of Sequential Decision Making (SDM) problems that are characterized by time-varying parameters. These parameter dynamics are either pre-specified or manipulable. At any given time instant the decision policy -- that governs the sequential decisions -- along with all the parameter values determines the cumulative cost incurred by the underlying SDM. Thus, the objective is to determine the manipulable parameter dynamics as well as the time-varying decision policy such that the associated cost gets minimized at each time instant. To this end we develop a control-theoretic framework to design the unknown parameter dynamics such that it locates and tracks the optimal values of the parameters, and simultaneously determines the time-varying optimal sequential decision policy. Our methodology builds upon a Maximum Entropy Principle (MEP) based framework that addresses the static parameterized SDMs. More precisely, we utilize the resulting smooth approximation (from the above framework) of the cumulative cost as a control Lyapunov function. We show that under the resulting control law the parameters asymptotically track the local optimal, the proposed control law is Lipschitz continuous and bounded, as well as ensure that the decision policy of the SDM is optimal for a given set of parameter values. The simulations demonstrate the efficacy of our proposed methodology.