Minimizing Maximum Regret in Commitment Constrained Sequential Decision Making

TL;DR

This paper introduces a method for multiagent planning where an agent commits to behaviors with probabilistic guarantees across multiple possible environments, adapting policies based on observations to minimize worst-case regret.

Contribution

It extends commitment-based planning to a worst-case setting with multiple environments, providing algorithms that adapt policies to observations to meet commitments and minimize maximum regret.

Findings

Algorithms ensure commitment satisfaction with history-dependent policies.

The approach minimizes maximum regret across possible environments.

Preliminary empirical results demonstrate effectiveness.

Abstract

In cooperative multiagent planning, it can often be beneficial for an agent to make commitments about aspects of its behavior to others, allowing them in turn to plan their own behaviors without taking the agent's detailed behavior into account. Extending previous work in the Bayesian setting, we consider instead a worst-case setting in which the agent has a set of possible environments (MDPs) it could be in, and develop a commitment semantics that allows for probabilistic guarantees on the agent's behavior in any of the environments it could end up facing. Crucially, an agent receives observations (of reward and state transitions) that allow it to potentially eliminate possible environments and thus obtain higher utility by adapting its policy to the history of observations. We develop algorithms and provide theory and some preliminary empirical results showing that they ensure an agent meets its commitments with history-dependent policies while minimizing maximum regret over the possible environments.

Tables

Table 1. Table 1: Max regret with varying horizon in Twin-States.

Horizon $T$	3	5	7	9	11	13
Greedy	5	5	9	13	17	21
MDPs-Best	3	7	13	19	25	31
MILP-CCL, $L=0$	3	6	10	15	19	22
MILP-CCL, $L=1,2$	1	3	6	8	9	11
MILP-CCL, $L\in [3,T]$	1	3	5	5	5	5
MILP-CCIL, $L=1$	1	3	5	5	5	5

Equations

$$h_t=⟨s_0,a_0,r_1,s_1,...,s_{t-1},a_{t-1},r_t,s_t⟩$$

$$\underset{\pi }{\mathrm{Pr}}(S_T\in \Phi |S_0=s_0;k)\ge p,\forall k\in {\kappa }_0.$$

$$U^{\pi }(k)=E_{\pi }\left[\sum _{t=0}^{T-1}{R}_{S_t{A}_t}(k)|S_0=s_0;k\right]$$

$$U_c^{\ast }(k)=\underset{\pi \in {\Pi }_c}{\max }{U}^{\pi }(k).$$

$${\rho }_c^{\pi }=\underset{k\in {\kappa }_0}{\max }{U}_c^{\ast }(k)-U^{\pi }(k).$$

$${\Pi }_c^{\ast }=\{\pi :\pi \in {\Pi }_c,{\rho }_c^{\pi }=\underset{{\pi }^{\prime }\in {\Pi }_c}{\min }{\rho }_c^{{\pi }^{\prime }}\}.$$

$$a_t=\underset{a}{\mathrm{argmin}}\underset{k\in {\kappa }_t}{\max }\{\underset{a^{\prime }}{\max }{R}_{s_t{a}^{\prime }}(k)-R_{s_{t}a}(k)\}.$$

$$\forall {\pi }^{\prime },\exists k\in {\kappa }_t\underset{{\pi }^{\prime }}{\mathrm{Pr}}(S_T\in \Phi |S_t=s_t;k)<p.$$

$$\underset{x}{\min }\underset{k}{\max }{U}_c^{\ast }(k)-U(k)$$

$$\mathrm{subject}\mathrm{to}$$

$$\forall kU(k)=\sum _{s,a}{x}_{sa}(k)R_{sa}(k)$$

$$\forall k,s,a{x}_{sa}(k)\ge 0$$

$$\forall k,s^{\prime }\sum _{a^{\prime }}{x}_{s^{\prime }{a}^{\prime }}(k)=\sum _{s,a}{x}_{sa}(k)P_{sa}^{s^{\prime }}(k)+{\delta }_{s^{\prime }{s}_0}$$

$$\forall k,k^{\prime },s,a\frac{x_{sa}(k)}{{\sum }_{a^{\prime }}{x}_{s{a}^{\prime }}(k)}=\frac{x_{sa}(k^{\prime })}{{\sum }_{a^{\prime }}{x}_{s{a}^{\prime }}(k^{\prime })}$$

$$\forall k\sum _{s\in \Phi }\sum _a{x}_{sa}(k)\ge p$$

$$\underset{\pi \in {\Pi }_c\cap {\Pi }_0}{\min }{\rho }_c^{\pi }.$$

$$x_{sa}^{\pi }(k):=E_{\pi }\left[\sum _{t=0}^{T-1}{1}_{\{S_t=s,A_t=a\}}|S_0=s_0;k\right].$$

$$\pi (a|s)=\frac{x_{sa}(k)}{{\sum }_{a^{\prime }}{x}_{s{a}^{\prime }}(k)}.$$

$$\pi (a|h_t)=\{\begin{array}{ll}\pi (a|b_t)& t<L\\ \pi (a|s_t,b_L)& t\ge L,\end{array}$$

$$\underset{\pi \in {\Pi }_c\cap {\Pi }_L}{\min }{\rho }_c^{\pi }.$$

$$\underset{x,y}{\min }\underset{k\in {\kappa }_0}{\max }{U}_c^{\ast }(k)-U(k)$$

$$\mathrm{subject}\mathrm{to}$$

$$\forall k\in {\kappa }_0$$

$$U(k)=\sum _{b\in {\mathcal{B}}_{[0,L)}^{b_0},a}{y}_{ba}(k){\tilde{R}}_{ba}(k)+\sum _{b_L\in {\mathcal{B}}_L^{b_0},s,a}{x}_{sa}^{b_L}(k)R_{sa}(k);$$

$$\forall k,b,a{y}_{ba}(k)\ge 0;$$

$$\forall k,b^{\prime }=⟨s^{\prime },{\kappa }^{\prime }⟩\in {\mathcal{B}}_{[0,L]}^{b_0}$$

$$\sum _{a^{\prime }}{y}_{b^{\prime }{a}^{\prime }}(k)=\sum _{b,a}{y}_{ba}(k){\tilde{P}}_{ba}^{b^{\prime }}(k)+{\delta }_{b^{\prime }{b}_0}{1}_{\{k\in {\kappa }^{\prime }\}};$$

$$\forall k,k^{\prime },b\in {\mathcal{B}}_{[0,L)}^{b_0},a\frac{y_{ba}(k)}{{\sum }_{a^{\prime }}{y}_{b{a}^{\prime }}(k)}=\frac{y_{ba}(k^{\prime })}{{\sum }_{a^{\prime }}{y}_{b{a}^{\prime }}(k^{\prime })};$$

$$\forall k,b_L\in {\mathcal{B}}_L^{b_0}{y}_{b_L}(k)=\sum _a{y}_{b_{L}a}(k);$$

$$\forall k,b_L\in {\mathcal{B}}_L^{b_0},s,a{x}_{sa}^{b_L}(k)\ge 0;$$

$$\forall k,b_L=⟨s_L,{\kappa }_L⟩\in {\mathcal{B}}_L^{b_0},s^{\prime }$$

$$\sum _{a^{\prime }}{x}_{s^{\prime }{a}^{\prime }}^{b_L}(k)=\sum _{s,a}{x}_{sa}^{b_L}(k)P_{sa}^{s^{\prime }}(k)+y_{b_L}(k)1_{\{k\in {\kappa }_L\}}{\delta }_{s^{\prime }{s}_L};$$

$$\forall b_L\in {\mathcal{B}}_L^{b_0},k,k^{\prime },s,a\frac{x_{sa}^{b_L}(k)}{{\sum }_{a^{\prime }}{x}_{s{a}^{\prime }}^{b_L}(k)}=\frac{x_{sa}^{b_L}(k^{\prime })}{{\sum }_{a^{\prime }}{x}_{s{a}^{\prime }}^{b_L}(k^{\prime })};$$

$$\forall k\in {\kappa }_0\sum _{b_L\in {\mathcal{B}}_L^{b_0}}\sum _{s\in \Phi ,a}{x}_{sa}^{b_L}(k)\ge p$$

$${\pi }_L(a|h_t)=\{\begin{array}{ll}{\pi }_L(a|b_t)=\frac{y_{b_{t}a}(k)}{{\sum }_{a^{\prime }}{y}_{b_t{a}^{\prime }}(k)}& t<L\\ {\pi }_L(a|s_t,b_L)=\frac{x_{s_{t}a}^{b_L}(k)}{{\sum }_{a^{\prime }}{x}_{s_t{a}^{\prime }}^{b_L}(k)}& t\ge L.\end{array}$$

$${\rho }_c^{{\pi }_L^{\ast }}\le {\rho }_c^{{\pi }_0^{\ast }}.$$

$${\rho }_c^{{\pi }_{L^{\prime }}^{\ast }}>{\rho }_c^{{\pi }_L^{\ast }},$$

$${\rho }_c^{{\pi }_{L^{\prime }}^{\ast }}\le {\rho }_c^{{\pi }_L^{\ast }},$$

$$p(k)=\underset{{\pi }_L^{\ast }}{\mathrm{Pr}}(S_T\in \Phi |S_L=s_L;k).$$

$$\underset{\pi }{\max }{E}_{\pi }\left[\sum _{t=L}^{T-1}{R}_{S_t{A}_t}(k)|S_L=s_L;k\right]$$

Full text

Introduction

When planning jointly, agents can benefit from making commitments to each other about what they will (or won’t) do that affects another agent, so that other agents can form their own plans accordingly. In the ideal case, commitments by an agent could allow the other agents to plan their behavior completely independently by relying on the commitments. For example, an agent could commit to free up a tool for another agent to use by a certain time and assuming that the only interaction among the two agents is the use of the tool, this can allow the other agent to plan independently.

Some existing computational models of commitments characterize them using formal logic (?; ?; ?; ?; ?; ?). When there is uncertainty about the consequences of actions, logical formulations associate conventions and protocols for managing such uncertainty (?; ?; ?). An alternative means of handling uncertainty, as in this paper, is to formalize commitments in decision-theoretic settings and explicitly allow for probabilistic guarantees of outcomes (?; ?; ?).

An interesting challenge in making and keeping commitments arises when the committing agent expects to learn information about its environment while executing its plan. What should a probabilistic commitment mean in such a setting? Recently Zhang et al. (?) provided an answer to this question in sequential decision problems where the committing agent interacts with an environment modeled as a controlled Markov process with a prior distribution over possible reward functions and has already made a probabilistic commitment to achieve a state at a certain time. The committing agent observes rewards while taking actions and thereby can refine its distribution over possible reward functions after each action. They formalize the meaning of a probabilistic commitment as requiring the agent to “execute a policy from the initial state that properly affects the committed state variables in expectation” (where this expectation is over both stochastic transitions and the effect of stochastic reward observations to the agent’s knowledge during plan execution).

Our main contributions in this paper are to extend the work of Zhang et al. to the worst-case non-Bayesian setting in which the agent knows that the sequential decision making task it is facing is from one of a set of Markov Decision Processes (MDPs), where both reward and transition dynamics could differ across MDPs, and nonetheless guarantees, at least, the same commitment probability in all MDPs. We propose a family of policy construction methods for the committing agent that adopts maximum regret as the performance criterion. We prove that policies constructed by the proposed methods respect this commitment semantics, and through experimental results we find they significantly outperform some baseline policies, such as the greedy policy that picks the next action minimizing myopic regret.

Example Domain

For illustrative purposes, we first present a two-state example, Twin-States, before we formalize the general problem. The Twin-States domain consists of two states with known deterministic transition dynamics but uncertain reward, as shown in Figure 1. The start state is A and the agent has three actions in each of the two states. Action $a_{0}$ moves the agent to the other state with no reward, while actions $a_{1}$ and $a_{2}$ keep it in the original state. Action $a_{1}$’s reward is 2 in state A and 3 in state B. Action $a_{2}$’s reward can be any element of the set $\{1,3,5\}$ in state A, and can be any element of set $\{0,2,4\}$ in state B. The agent commits to being in state A at the time horizon with probability one.

Action $a_{2}$ could be more rewarding than action $a_{1}$, or less. If the horizon is large enough, the agent has enough time to figure out the reward of action $a_{2}$ in both states, and depending on the observed rewards it will thereafter have a clear preference for one state-action. This behavior has two interesting properties. First, it chooses actions based on the previous observations (i.e., history). Reward for actions that the agent has already taken helps it choose actions wisely in the future. Second, it is constrained by the commitment. At the time step just before the time horizon in the Twin-States domain, if the agent is in state B it should take action $a_{0}$, otherwise it is in state A and should not take action $a_{0}$. In our experimental work below, we will show how our proposed methods can solve the Twin-States problem to compute such policies.

Problem Formulation

We consider settings in which an agent knows that its sequential decision making problem is one out of $K$ possible MDPs but does not know which MDP it is in at the start. We assume that all $K$ MDPs have the same state and action spaces but possibly different transition and reward functions. During execution, the agent can observe the state and the reward and this can provide information about the MDP it is in. The environment is formally defined by the tuple $\mathcal{E}=\langle\mathcal{S},\mathcal{A},\{P(k),R(k)\}_{k=1}^{K},s_{0}\rangle$, where $\mathcal{S}$ and $\mathcal{A}$ are finite environment state (henceforth env-state) and action spaces, respectively, and $s_{0}\in\mathcal{S}$ is the initial env-state. If the agent is in MDP $k$, then on taking action $a\in\mathcal{A}$ in env-state $s\in\mathcal{S}$ the agent receives reward $r=R_{sa}(k)$ and the environment transitions to env-state $s^{\prime}$ with probability $P^{s^{\prime}}_{sa}(k)$. Throughout we assume that the planning horizon is finite, which in turn implies that time is part of the env-state. Let $\mathcal{S}_{t}$ be the set of env-states at time step $t$, $S_{t}$ be a random variable indicating the env-state at time step $t$ whose specific realization is denoted $s_{t}$, $A_{t}$ be a random variable indicating the action taken at time step $t$, whose specific realization is denoted $a_{t}$, and let

[TABLE]

be the history at time step $t$. Because of the agent’s lack of knowledge about the MDP it is facing, we consider history-dependent stochastic policies and use $\pi(a|h)$ to denote the probability of choosing action $a$ given history $h$ under policy $\pi$. During execution, history gives the agent knowledge about the true MDP it might be in or, equivalently, the MDPs that it cannot be in. Formally, we can summarize the current history $h$ into a knowledge state, $b:=\langle s,\kappa\rangle$, where $s$ is the current env-state, and $\kappa:=\{k:k\sim h\}$ is the set of indices of MDPs consistent with $h$. Initially, the agent is in knowledge state $b_{0}=\langle s_{0},\kappa_{0}\rangle$ where $\kappa_{0}=\{1,2...K\}$. Let $B_{t}$ be a random variable indicating the knowledge state at time step $t$, and $b_{t}$ be the knowledge state given history $h_{t}$. (In general there is a many to one mapping from histories to knowledge states.) We define the agent’s planning objective below.

Commitment Semantics

Note that there are two types of uncertainty in our setting. There is non-probabilistic uncertainty (i.e., incomplete knowledge) over which MDP the agent is facing, and there is probabilistic uncertainty (i.e., stochastic state transitions and possibly rewards) within an MDP.

Our commitment in this uncertain environment $\mathcal{E}$ is formally defined as follows.

Definition 1**.**

A probabilistic commitment $c$ is formally defined as a tuple $\langle\Phi,T,p\rangle$, where $\Phi\subset\mathcal{S}$ is the commitment env-state space, $T$ is the commitment finite time horizon, and $p$ is the commitment probability. By making commitment $c$, the agent is constrained to follow a policy $\pi$, such that

[TABLE]

From Equation (2), the semantics of a probabilistic commitment is clear: the agent is constrained to follow a (in general history-dependent) policy, such that starting at the initial env-state, it will reach a env-state in the committed env-state space, $S_{T}$, at the time horizon, $T$, with at least the committed probability, $p$, no matter which MDP it is in. Given probabilistic commitment $c$, let $\Pi_{c}$ be the set of all history-dependent stochastic policies that satisfy Equation (2).

Minimax Regret

In this paper, we are interested in finding a good policy given a probabilistic commitment, using maximum regret as the performance criterion. Let

[TABLE]

be the expected cumulative reward under policy $\pi$ if the true MDP is $k$, and let $U^{*}_{c}(k)$ be the expected cumulative reward under the optimal policy respecting the semantics of commitment $c$ if the true MDP is $k$:

[TABLE]

Finding $U^{*}_{c}(k)$ amounts to solving a standard constrained MDP problem and this can be done efficiently by linear programming (?). Given commitment $c$, let $\rho_{c}^{\pi}$ denote the maximum regret of policy $\pi$ under $c$, i.e.,

[TABLE]

Let $\Pi^{*}_{c}$ be the set of policies that minimizes the maximum regret while respecting the commitment semantics,

[TABLE]

The agent’s planning goal is to find a policy in $\Pi^{*}_{c}$. We conclude this section with a series of formal observations showing that straightforward planning methods will not be enough to construct policies in $\Pi^{*}_{c}$.

Observation 1 says that in general it is not sufficient to search over policies that are optimal for some MDP.

Observation 1**.**

Let $\pi^{*}_{c}(k)=\operatorname*{arg\,max}_{\pi\in\Pi_{c}}U^{\pi}(k)$ be a policy respecting commitment $c$ that is optimal if the true MDP is $k$. Then, in general we have $\pi^{*}_{c}(k)\not\in\Pi^{*}_{c},\forall k$.

Observation 2 says that in general it is not sufficient to greedily pick the next action that minimizes the maximum myopic regret.

Observation 2**.**

Let $\pi_{G}$ be the greedy policy under which the agent selects the next action that minimizes the maximum myopic regret over the possible MDPs consistent with the current knowledge, i.e.

[TABLE]

Then, in general we have $\pi_{G}\not\in\Pi^{*}_{c}$.

Observation 3 says that it is possible that no policy in $\Pi^{*}_{c}$ is deterministic even if all MDPs in the environment are deterministic.

Observation 3**.**

There exists an environment $\mathcal{E}$ where all MDPs are deterministic, i.e. $\forall k,s,a~{}\exists s^{\prime}$ such that $P_{sa}^{s^{\prime}}(k)=1$, and no policy in $\Pi^{*}_{c}$ is deterministic.

The Twin-States domain provides a proof of the above observations by example as we verify in the Section on Empirical Results below.

Finally, we might think whenever the agent learns more about the true MDP during execution it is a good idea to re-plan from the current env-state with the original commitment probability. Clearly, if during execution one can always find a policy that achieves the original commitment probability conditioned on the current env-state, such a re-planning approach will certainly respect the commitment semantics. Observation 4 says that this is not always possible, and the example shown in Figure 2 verifies it.

Observation 4**.**

There exists $\pi\in\Pi_{c}$ such that if the agent executes policy $\pi$ for the first $t>0$ time steps starting in state $s_{0}$, the history generated $h_{t}$ is such that

[TABLE]

Methods

In this section we introduce several methods for constructing policies that respect the commitment semantics for a given commitment $c$.

Commitment Constrained No-Lookahead

Let $\Pi_{0}$ be the set of all Markov policies, i.e., policies that choose actions solely as a function of the current env-state (and ignore $\kappa$). Assuming $\Pi_{0}\cap\Pi_{c}\neq\emptyset$, our Commitment Constrained No-Lookahead (CCNL) method of Figure 3 finds a minimax regret Markov policy respecting the commitment semantics, which is a solution to the following problem:

[TABLE]

For MDP $k$, each policy $\pi$ has a corresponding occupancy measure $x^{\pi}(k)$ for env-state-action pairs:

[TABLE]

We will use shorthand notation $x(k)$ in place of $x^{\pi}(k)$ when policy $\pi$ is clear from the context. If $\pi$ is a Markov policy, it can be recovered from its occupancy measure via

[TABLE]

Figure 3 presents our straightforward adaptation of the linear program for finding constrained-optimal policies in MDPs (see the caption of Figure 3 for details).

Commitment Constrained Lookahead

During execution, the agent can observe the env-state transitions and reward and reason about the true MDP it might be in or, equivalently, the MDPs that it cannot be in. Thus, restricting the agent to Markov policies as in the previous section will lead to larger regret than is necessary. Here we consider the general case where the agent may choose actions based on the knowledge state (or equivalently history) for the first $0<L\leq T$ steps, and use the env-state for the remaining time steps (if $L=0$, we recover the Markov policy case above). We refer to $L$ as the knowledge-state-update boundary. The resulting $L$-updates policy has the form:

[TABLE]

where $b_{t}$ is the knowledge state consistent with $h_{t}$ and $b_{L}$ is the knowledge state consistent with $h_{L}$ when $t\geq L$. It is important to note that after the knowledge-state-update boundary, the policy conditions on both the env-state as well as the last updated knowledge state $b_{L}$.

For example, Figure 4 shows a $(L=)1$-updates policy constructed in the Twin-States domain. After taking some action in the initial knowledge state, depending on which knowledge state it actually ends up in at time $L=1$, it then executes a Markov policy, represented by a curve, all the way up to the horizon. Those Markov policies starting from time step $L=1$ are not necessarily the same, which gives the agent flexibility of choosing different behaviors based on the last-updated knowledge about the environment.

Let $\Pi_{L}$ be the set of all $L$-updates policies. Our Commitment Constrained Lookahead (CCL) method finds a minimax regret $L$-updates policy respecting the commitment semantics, which is a solution to the following problem:

[TABLE]

Problem (6) can be expressed by the program in Figure 5.

The program in Figure 5 introduces as decision variables $y(k)$ and $x(k)$ for every possible MDP $k$, where $y(k)$ is the knowledge state-action occupancy measure if the true MDP is $k$, but only for those knowledge states reachable within the first $L$ time steps, and $x(k)$ is the env-state-action occupancy measure for the env-states in the remaining $T-L$ time steps if the true MDP is $k$. See the caption of Figure 5 for details.

Any $L$-updates policy $\pi_{L}$ respecting the commitment semantics can be derived from a feasible solution to the program in Figure 5 via

[TABLE]

Theorem 1 states that CCL with knowledge-state-update boundary $L$ finds a minimax regret policy in $\Pi_{c}\cap\Pi_{L}$.

Theorem 1**.**

If $\Pi_{c}\cap\Pi_{L}\neq\emptyset$ holds for commitment $c$, the program in Figure 5 is feasible. Let $x^{*},y^{*}$ be its optimal solution, then the policy derived via Equation (8) with $x^{*},y^{*}$ is a minimax regret policy in $\Pi_{c}\cap\Pi_{L}$.

The proofs for Theorem 1 and the theorems that follow are presented in the Appendix of a full paper available on arXiv.

Intuitively, a knowledge-state-update boundary greater than zero may help the agent choose actions according to its changing knowledge about the actual MDP it is in and therefore improve the performance. Theorem 2 says the maximum regret of the policy derived by CCL using any $L>0$ is upper bounded by the maximum regret of the policy derived by CCNL.

Theorem 2**.**

If $\Pi_{c}\cap\Pi_{0}\neq\emptyset$ holds for commitment $c$, the program in Figure 5 is feasible for any $L\in[0,T]$. Let $\pi_{L}^{*}$ be the policy derived by CCL using knowledge-state-update boundary $L$, then for any $L\in[0,T]$ we have

[TABLE]

However, one has to be careful in using deeper boundaries because the performance of CCL is guaranteed to be monotonically non-decreasing in $L$ only when transition dynamics is invariant across MDPs, but this monotonicity cannot be guaranteed in general, as stated in Theorem 3 and Theorem 4.

Theorem 3**.**

There exists an environment $\mathcal{E}$, a commitment $c$, $L^{\prime}>L>0$ satisfying $\Pi_{c}\cap\Pi_{L}\neq\emptyset$ and $\Pi_{c}\cap\Pi_{L^{\prime}}\neq\emptyset$, such that

[TABLE]

where $\pi_{L}^{*}$ and $\pi_{L^{\prime}}^{*}$ are the policies derived by CCL using boundaries $L$ and $L^{\prime}$, respectively.

Theorem 4**.**

If the transition dynamics does not vary across MDPs in environment $\mathcal{E}$, i.e. $\forall k,k^{\prime},P(k)=P(k^{\prime})$, and $\Pi_{c}\cap\Pi_{L}\neq\emptyset$ for boundary $L$. Then for any $L^{\prime}>L$ we have $\Pi_{c}\cap\Pi_{L^{\prime}}\neq\emptyset$, and

[TABLE]

where $\pi_{L}^{*}$ and $\pi_{L^{\prime}}^{*}$ are the policies derived by CCL using boundaries $L$ and $L^{\prime}$, respectively.

Commitment Constrained Iterative Lookahead

Commitment Constrained Iterative Lookahead (CCIL), as the name suggests, iteratively applies the CCL technique during execution. Suppose starting from the initial knowledge state the agent executes the first $L$ actions prescribed by a minimax regret $L$-updates CCL policy $\pi^{*}_{L}$ derived by solving the program in Figure 5 and ends up in knowledge state $b_{L}\in\mathcal{B}_{L}^{b_{0}}$. Instead of executing the remaining $T-L$ actions prescribed by $\pi^{*}_{L}$, the agent can re-construct a new $L$-updates policy with an initial knowledge state now $b_{L}$. This policy reconstruction is helpful because the agent gets more knowledge about the true MDP by observing the transitions and reward in the first $L$ steps. Due to the changed initial knowledge state, naively sticking with the original commitment probability might lead to the difficulty stated in Observation 4. To respect the commitment semantics, the agent should instead plan with a commitment probability updated as follows. Let $b_{L}=\langle s_{L},\kappa_{L}\rangle$, where $s_{L}$ is the current env-state, and $\kappa_{L}$ is the set of MDPs consistent with the history up to time step $L$. For every possible MDP $k\in\kappa_{L}$, update the commitment probability as the achieved probability if the agent were to stick with $\pi^{*}_{L}$ from $s_{L}$:

[TABLE]

Then, the agent can construct a new $L$-updates policy by solving the program in Figure 5 with the following modifications:

Start from current knowledge state $b_{L}$ instead of $b_{0}$, i.e. replace every $b_{0}$ with $b_{L}$, and $\kappa_{0}$ with $\kappa_{L}$ in the program. 2. 2.

Plan with the updated commitment probabilities, i.e. replace $p$ in the last constraint of the program with $p(k)$ calculated as Equation (9). 3. 3.

Replace $U^{*}_{c}(k)$ with $U^{*}_{s_{L},p(k)}(k)$ which is defined as the optimal objective value of the following problem:

[TABLE]

which is the expected cumulative reward of the optimal policy that achieves commitment probability $p(k)$ from current env-state $s_{L}$ in MDP $k$.

This modified program is guaranteed to be feasible because the original $L$-updates policy $\pi^{*}_{L}$ itself is a solution. CCIL iteratively applies the above procedure every $L$ steps. We outline CCIL in Algorithm 1, and Theorem 5 formally states that it respects our commitment semantics.

Theorem 5**.**

If $\Pi_{c}\cap\Pi_{L}\neq\emptyset$ holds for commitment $c$ and boundary $L>0$, let $\pi^{\mathrm{IL}}_{L}$ be the history-dependent policy defined as Algorithm 1. We have $\pi^{\mathrm{IL}}_{L}\in\Pi_{c}$, i.e., CCIL respects the commitment semantics.

MILP Formulation

The CCL program in Figure 5 introduces quadratic equality constraints (7e) and (7i) to ensure that the action selection rules derived from occupancy measures in all possible MDPs are identical. These constraints make the optimization problem non-convex and hard to solve. In practice, many math-programming solvers are unable to handle programs with quadratic equality constraints. Although some solvers can deal with such programs, they often need to take as input a feasible solution as the starting point, but finding a feasible solution by itself might be difficult, and the final solutions are usually sensitive to starting points. Here we introduce a straightforward modification to the CCL program in Figure 5 that replaces the quadratic equality constraints with mixed integer constraints, and therefore reformulates it into a Mixed Integer Linear Program (MILP) that has many available solvers. The cost of this reformulation is that the derived policy is restricted to be deterministic.

Specifically, we introduce indicators $\Delta$ into the CCL program in Figure 5 as additional decision variables with the following constraints:

[TABLE]

Then, any feasible solution with the above constraints replacing constraints (7e) and (7i) of the program in Figure 5 yields a deterministic policy via Equation (8), which can be alternatively expressed using the indicator variables:

[TABLE]

Note that the objective function of the program in Figure 5 is non-linear due to the max operator. However, it is easy to reformulate it into a linear objective function with a set of linear constraints. In particular, one can introduce a scalar variable $z$ to replace the objective function (7a) with

[TABLE]

and add the following constraints on $z$

[TABLE]

With the above modifications, the program in Figure 5 becomes a MILP. The derived policy via (11) using an optimal solution to this MILP is a deterministic policy that minimizes the maximum regret of all deterministic policies in $\Pi_{c}\cap\Pi_{L}$ (assuming this intersection is non-empty).

Empirical Results

We evaluate the performance of CCL and CCIL, under various choices of the boundary $L$, first on the Twin-States domain of Figure 1 that has uncertain rewards, and second on the Slippery T-Maze gridworld domain of Figure 7 that has uncertain transition dynamics. CCL and CCIL MILP programs are solved using CPLEX 12.6.

Results on the Twin-States Domain

The main goals of the experiments on this domain are

1. to provide a constructive proof of Observations 1 to 3,
2. to evaluate the loss of the MILP formulation in a domain where an exact stochastic CCL policy can be computed, and
3. to compare the performance of CCL and CCIL using various boundaries against simple policy construction methods.

Short horizon. Here we set the time horizon to two so that we can find an exact stochastic minimax regret CCL policy111This exact policy is found not by solving the program in Figure 5 but as follows. Note that with only two actions available, the agent should not move to state B because it has to move back to state A using the second action and will get no reward at all. We exploit this fact to compute an exact stochastic CCL policy, i.e. an exact solution to the program in Figure 5 by solving another equivalent mathematical program:

1. Introduce $\pi(a|b),\pi(a|s;b_{L})$ as decision variables, which are the probability of choosing action $a$ under an $L$-updates policy for $t<L$ and $t\geq L$, respectively. Because choosing $a_{0}$ is sub-optimal, we need to only consider $a\in\{a_{1},a_{2}\}$.
2. Express the maximum regret as the objective function, the only constraint is that $\pi(\cdot|b),\pi(\cdot|s;b_{L})$ should be valid probability measures. The commitment semantics is automatically satisfied because we don’t need to include action $a_{0}$. and compare it with that found using the MILP formulation.

Figure 6 plots the maximum regret under various choices of boundary $L$ using exact CCL, MILP-CCL, and MILP-CCIL. Because exact CCL achieves better performance than MILP-CCL, it is clear that the derived policy must be stochastic, which provides a constructive proof of Observation 3.

Longer horizon. Here we are concerned with comparing MILP-CCL and MILP-CCIL against the following baseline policy construction methods mentioned in Observation 1 and Observation 2 under longer than 2 time horizon.

• •

MDPs-Best: First find the optimal policies respecting the commitment semantics for every possible MDP, i.e. $\pi^{*}_{k}=\operatorname*{arg\,max}_{\pi\in\Pi_{c}}U^{\pi}(k)$. The MDPs-Best policy is the one out of $\{\pi^{*}_{k}\}_{k=1}^{K}$ that minimizes the maximum regret.

• •

Greedy: Select the next action that minimizes the maximum one-myopic regret over the possible MDPs consistent with the current history, i.e.

[TABLE]

where $\mathcal{A}_{t}$ is the set of actions available at time $t$ that are chosen to guarantee the commitment semantics is respected. For this domain, we let $\mathcal{A}_{t}=\{a_{0},a_{1},a_{2}\}$ if $t<T-1$. When $t=T-1$, i.e. for the last action, $\mathcal{A}_{t}=\{a_{0}\}$ if $s_{t}$ is B, or $\mathcal{A}_{t}=\{a_{1},a_{2}\}$ if $s_{t}$ is A.

Table 1 summarizes the results. For MILP-CCL, performance is monotonic. It takes three steps to resolve the reward uncertainty by taking action $a_{2}$ in state A, moving to state B, and then taking action $a_{2}$ again. This explains why $L$ larger than three does not improve the performance. If the horizon is large enough, the agent should explore the reward of action $a_{2}$ in both states, then execute the action with the highest reward before going back to state A to respect the commitment semantics. We find that is exactly what MILP-CCL($L\in[3,T]$) and MILP-CCIL($L=1$) do when horizon $T\geq 7$, which causes a max regret of 5 when reward of $a_{2}$ is the lowest (i.e., 1 in state A and 0 in state B).

Results on the Slippery T-Maze

The main goals of the experiments reported here are to evaluate CCL and CCIL with the MILP formulation in a domain where

1. the transition dynamics are uncertain, and
2. the commitment probability is less than one, and thus stochastic action selection is more likely to be crucial to achieving better performance.

The domain consists of two corridors that are connected as shown in Figure 7. The agent starts in the cell with a black dot and can move in four directions. Staying in cell “r” results in a positive unit of reward every time step, but the agent commits to being in cell “c” at the time horizon. There are an uncertain number of consecutive slippery cells between cell “s” and the black dot cell. In a slippery cell movement actions succeed with probability .8. Cell “s” is known to be slippery. The agent does not know in advance the number of slippery cells, which makes the transition dynamics uncertain.

Figure 8 shows the results under commitment time horizon $T=10$ and commitment probability $p=0.6$. The maximum regret of MILP-CCL is equal to the objective value of the mathematical program, which can be directly obtained, while the performance of MILP-CCIL is estimated by averaging many simulated episodes. The latter is seen to achieve better maximum regret than the former for low values of $L$. Interestingly, and perhaps unexpectedly, unlike for the Twin-States domain, the performance of MILP-CCL is not monotonic in boundary $L$. The explanation lies in the fact that though the MILP-CCL policy is a deterministic function of history, the part of the policy that occurs after the boundary $L$ when viewed as a function of env-state alone is stochastic. This is because the knowledge-state at time $L$ is stochastic due to the stochastic transition dynamics (recall that the policy after $L$ is allowed to condition on the knowledge state at time $L$). Thus if $L$ is too large, the agent cannot take advantage of this stochasticity and suffers larger regret than for intermediate values of $L$. On the other hand if $L$ is too small, then the knowledge-state at $L$ is not informative enough to be helpful. Also interestingly, MILP-CCIL can take advantage of this implicit stochasticity using smaller $L$. However, when $L$ is large, MILP-CCIL achieves the same poor performance as MILP-CCL, because when $L$ is large the agent is likely to be in the vertical corridor where it no longer gets new knowledge about how many slippery cells there are and therefore iterative lookahead does not help.

Conclusion

In this paper we developed a commitment semantics for achieving a specific state by a certain time with at least a certain probability in environments that have non-probabilistic uncertainty about the possible MDP the committing agent is facing as well as probabilistic uncertainty about the consequences of actions (within the true MDP). Our Commitment Constrained Lookahead (CCL) family of algorithms plan (offline) low-regret policies respecting the commitment semantics. We provided analysis and empirical results on the impact of the knowledge-state-update boundary, which is an input-parameter to CCL, on the performance of the planned policy. We extended CCL to Commitment Constrained Iterative Lookahead (CCIL), which is an iterative algorithm that adjusts the policy online. Exact CCL and CCIL require solving non-convex programs and thus we also introduced a MILP formulation that restricts the agent to deterministic policies. Our empirical results indicate that the MILP versions of both CCL and CCIL outperform baseline methods, and that CCIL is more robust than CCL.

Acknowledgments This work was supported in part by the Air Force Office of Scientific Research under grant FA9550-15-1-0039. Any opinions, findings, conclusions, or recommendations expressed here are those of the authors and do not necessarily reflect the views of the sponsors.

Technical Proofs

Proof of Theorem 1.

We need to show any policy in $\Pi_{c}\cap\Pi_{L}$ one-to-one maps to a feasible solution to the program in Figure 5.

For any policy $\pi\in\Pi_{c}\cap\Pi_{L}$, we are going to define vectors $m(\pi)$ and $n(\pi)$ such that they satisfy the constraints of the program in Figure 5 if treated as the decision variables $x$ and $y$, respectively. Given any policy $\pi\in\Pi_{c}\cap\Pi_{L}$, let $n(\pi,k)$ be its knowledge state-action occupancy measure if the true MDP is $k$ for knowledge states in $\mathcal{B}_{[0,L]}^{b_{0}}$, and $m(\pi,k)$ be its env-state-action occupancy measure for env-states from time step $L$ on.

[TABLE]

where $t$ is the time of knowledge state $b$. We next show $n_{ba}(\pi,k)$ satisfies the constraints if treated as $y_{ba}(k)$ and $m^{b_{L}}_{ba}(\pi,k)$ satisfies the constraints if treated as $x^{b_{L}}_{ba}(k)$.

If treated as $y_{ba}(k)$, $n_{ba}(\pi,k)$ satisfies constraint (7c) because $n_{ba}(\pi,k)\geq 0$.

If treated as $y_{ba}(k)$, $n_{ba}(\pi,k)$ satisfies constraint (7d) because if $b^{\prime}=b_{0}$, for the LHS of (7d) we have

[TABLE]

and for the RHS of (7d) we have

[TABLE]

If $b^{\prime}\in\mathcal{B}^{b_{0}}_{[0,L]}\setminus\{b_{0}\}$, for the LHS of (7d) we have

[TABLE]

and for the RHS of (7d) we have

[TABLE]

If treated as $y_{ba}(k)$, $n_{ba}(\pi,k)$ satisfies constraint (7d) because

[TABLE]

and therefore for any $k$

[TABLE]

which is independent of $k$.

Similarly one can show that $m^{b_{L}}_{ba}(\pi,k)$ satisfies constraints (7g), (7h), and (7i) if treated as $x^{b_{L}}_{ba}(k)$.

If treated as $x^{b_{L}}_{ba}(k)$, $m^{b_{L}}_{ba}(\pi,k)$ also satisfies constraint (7j) because $\pi\in\Pi_{c}$.

Constraints (7b) and (7f) are naturally satisfied because they are defining new variables.

Now for the other direction, given a feasible solution $x,y$ to the program, let policy $\pi$ be the derived policy via (8). Then $\pi$ is in $\Pi_{L}$ by definition. Further we have $m^{b_{L}}_{sa}(\pi,k)=x^{b_{L}}_{sa}(k),n_{ba}(\pi,k)=y_{ba}(k)$, and therefore $\pi$ is also in $\Pi_{c}$ because $x$ satisfies commitment constraint (7j). ∎

Proof of Theorem 2.

Because CCL with boundary $L$ finds a minimax regret policy in $\Pi_{c}\cap\Pi_{L}$, it is sufficient to show

[TABLE]

This holds because given any Markov policy $\pi_{0}\in\Pi_{0}$, we can define an $L$-updates policy $\pi_{L}\in\Pi_{L}$ that is equivalent to $\pi_{0}$

[TABLE]

Thus, we know that $\pi_{0}\in\Pi_{L}$. ∎

We first prove Theorem 4 and then Theorem 3.

Proof of Theorem 4.

It’s sufficient to show that the statement holds when $L^{\prime}=L+1$. We next show that given any policy $\pi_{L}\in\Pi_{L}$, there exists an $(L+1)$-updates policy, $\pi_{L+1}$, that mimics $\pi_{L}$ when $P(k)=P(k^{\prime})~{}\forall k,k^{\prime}$, and therefore $\rho_{c}^{\pi_{L+1}^{*}}\leq\rho_{c}^{\pi_{L}^{*}}$.

For the first $L$ actions, an $(L+1)$-updates policy can map the current knowledge state to a distribution of the next actions identical to $\pi_{L}$. The action that is going to take at time step $L$ by $\pi_{L}$ can also be recovered by an $(L+1)$-updates policy, which gives

[TABLE]

If $P(k)=P(k^{\prime})~{}\forall k,k^{\prime}$, then $\pi_{L+1}$ can also recover $\pi_{L}$ for $t\geq L+1$. To see this, note if $P(k)=P(k^{\prime})~{}\forall k,k^{\prime}$ we have

[TABLE]

Therefore, under any $L$-updates policy $\pi_{L}$ and conditioned on being in knowledge state $b_{L+1}$ at time step $L+1$, the agent thereafter selects actions according to $\pi_{L}(\cdot|s_{t},b_{L})$ with probability defined in Equation (12) that does not depend on $k$. Because the transition function is known, the occupancy measure of $\pi_{L}$ for $t\geq L+1$ conditioned on any $b_{L+1}$ can be achieved by a stochastic Markov policy from $b_{L+1}$, which can be expressed by an $(L+1)$-updates policy as $\pi_{L+1}(\cdot|s_{t},b_{L+1})$. ∎

Proof of Theorem 3.

In general $P(k)=P(k^{\prime})~{}\forall k,k^{\prime}$ does not hold, and therefore condition (12) in the proof of Theorem 4 does not hold. Let $b_{L+1}=\langle s_{L+1},\kappa_{L+1}\rangle$ be a knowledge state at time step $L+1$.

Inspired by this we now give an example as a formal constructive proof. In this example the env-state space is $\{0,1,2,3\}$ and let state 0 be the initial state. There are two actions $a_{0},a_{1}$ and $K=2$ possible MDPs. The time horizon is three and the commitment probability is zero. The transition dynamics and reward of these MDPs are as follows.

In MDP $k=1$, $\Pr(1|0,a;k=1)=0.9$, $\Pr(2|0,a;k=1)=0.1$, $\Pr(3|1,a;k=1)=\Pr(3|2,a;k=1)=1.0$ for both $a=a_{0},a_{1}$. State 3 is an absorbing state. Doing $a_{0}$ in state 3 gives a positive unit of reward. There’s no reward elsewhere. 2. 2.

In MDP $k=2$, $\Pr(1|0,a;k=1)=0.1$, $\Pr(2|0,a;k=1)=0.9$, $\Pr(3|1,a;k=1)=\Pr(3|2,a;k=1)=1.0$ for both $a=a_{0},a_{1}$. State 3 is an absorbing state. Doing $a_{1}$ in state 3 gives a positive unit of reward. There’s no reward elsewhere.

The maximum regret when $L=2$ is 0.5, but $L=1$ can achieve a maximum regret of 0.1. ∎

Proof of Theorem 5.

We need to show

[TABLE]

Let $\pi_{L}$ be the CCL $L$-updates policy derived from the program in Figure 5. For any $k\in\kappa_{0}$, we can calculate the achieved commitment probability of $\pi^{\mathrm{IL}}_{L}$ by conditioning on the knowledge state it will visit at time $L>0$,

[TABLE]

The second equality holds because $\pi^{\mathrm{IL}}_{L}$ is identical to $\pi_{L}$ in the first $L$ steps. The first inequality holds because CCIL iteratively applies $L$-step lookahead in Algorithm 1 line 15 with the commitment probability achieved by the policy of the previous iteration calculated in line 12. The modified program in Algorithm 1 line 15 is always feasible because $\Pi_{0}\subset\Pi_{L}$ which is shown in the proof of Theorem 2. ∎