Synthesis of Provably Correct Autonomy Protocols for Shared Control

TL;DR

This paper presents a formal framework for synthesizing shared control protocols that ensure safety and performance in robotic systems, accounting for human behavior uncertainty, using probabilistic logic and inverse reinforcement learning.

Contribution

It introduces a novel approach combining MDP modeling, inverse reinforcement learning, and quasiconvex programming to synthesize provably correct shared control strategies.

Findings

Successfully applied to wheelchair navigation and UAV planning.

Ensures safety and performance with probabilistic guarantees.

Produces human-like behavior in control strategies.

Abstract

We synthesize shared control protocols subject to probabilistic temporal logic specifications. More specifically, we develop a framework in which a human and an autonomy protocol can issue commands to carry out a certain task. We blend these commands into a joint input to a robot. We model the interaction between the human and the robot as a Markov decision process (MDP) that represents the shared control scenario. Using inverse reinforcement learning, we obtain an abstraction of the human's behavior and decisions. We use randomized strategies to account for randomness in human's decisions, caused by factors such as complexity of the task specifications or imperfect interfaces. We design the autonomy protocol to ensure that the resulting robot behavior satisfies given safety and performance specifications in probabilistic temporal logic. Additionally, the resulting strategies generate behavior as similar to the behavior induced by the human's commands as possible. We solve the underlying problem efficiently using quasiconvex programming. Case studies involving autonomous wheelchair navigation and unmanned aerial vehicle mission planning showcase the applicability of our approach.

Tables

Table 1. TABLE I: Scalability results for the gridworld example. We list the synthesis time of the both approaches in seconds. ’ δ G subscript 𝛿 G \delta_{\textnormal{G}} ’ and δ QCP subscript 𝛿 QCP \delta_{\textnormal{QCP}} ’ refer to the maximal deviation of the greedy and QCP approach.

Gridworld size	Number of states	Number of transitions	Synthesis time with the   greedy approach (sec)	${\delta }_{\text{G}}$	Synthesis time with the   QCP approach (sec)	${\delta }_{\text{QCP}}$
$8\times 8$	$2,304$	$36,864$	$14.12$	$0.145$	$31.49$	$0.031$
$10\times 10$	$3,600$	$57,600$	$23.80$	$0.231$	$44.61$	$0.042$
$12\times 12$	$14,400$	$230,400$	$250.78$	$0.339$	$452.27$	$0.050$
$20\times 20$	$40,000$	$640,000$	$913.23$	$0.373$	$1682.05$	$0.048$

Table 2. TABLE II: Results for different specification thresholds for the probability and expected time in the AMASE example. ’ β 𝛽 \beta ’ and ’ κ 𝜅 \kappa ’ refer to the threshold for the probability and the expected time of the specification.

$\beta$	$\kappa$	Synthesis time with the   QCP approach (sec)	${\delta }_{\text{QCP}}$
$0.7$	$20$	$827.37$	$0.380$
$0.7$	$40$	$749.14$	$0.126$
$0.7$	$80$	$722.81$	$0.054$
$0.9$	$20$	$888.29$	$0.598$
$0.9$	$40$	$795.98$	$0.163$
$0.9$	$80$	$732.41$	$0.100$

Table 3. TABLE III: Results for different perturbations of the human strategy in the AMASE example. ’ δ max subscript 𝛿 max \delta_{\textnormal{max}} ’ refers to the maximal perturbation introduced to the human strategy. δ ap subscript 𝛿 ap \delta_{\textnormal{ap}} refers to the maximal deviation between the repaired strategy and the human strategy.

$\beta$	${\delta }_{\text{max}}$	${\delta }_{\text{ap}}$	Synthesis time with the   QCP approach (sec)	${\delta }_{\text{QCP}}$
$0.7$	$0.1$	$0.170$	$725.93$	$0.032$
$0.7$	$0.2$	$0.274$	$718.14$	$0.036$
$0.7$	$0.5$	$0.506$	$696.60$	$0.037$
$0.9$	$0.1$	$0.270$	$732.54$	$0.091$
$0.9$	$0.2$	$0.345$	$745.05$	$0.092$
$0.9$	$0.5$	$0.534$	$798.01$	$0.101$

Equations

$${\mathcal{P}}^{\sigma }(s,s^{\prime })=\sum _{\alpha \in Act(s)}\sigma (s,\alpha )\cdot \mathcal{P}(s,\alpha ,s^{\prime })\text{mbox}foralls,s^{\prime }\in S.$$

$$x_{\sigma }(s,\alpha )=\mathbb{E}\left[\sum _{t=0}^{\infty }P({\alpha }_t=\alpha |s_t=s)\right],$$

$${\mathbb{P}}_{{\mathcal{M}}^{\sigma }}(s_I\models \phi )={\mathbb{P}}_{{\mathcal{M}}^{\sigma }}\{{\varrho }^{\sigma }\in Pat{h}^{\sigma }(\mathcal{M}):\mathcal{L}({\varrho }^{\sigma })\models \phi \}$$

$${\mathbb{P}}_{{\mathcal{M}}^{\sigma }}(s_I\models \phi )\ge \beta ,$$

$$\sum _{s^{\prime }\in C}{\mathcal{P}}_p(s,D(s),s^{\prime })=1,$$

$$\underset{{\sigma }_{ha}\in {Str}^{\mathcal{M}}}{\mathrm{minimize}}$$

$${\mathbb{P}}_{{\mathcal{M}}^{{\sigma }_{ha}}}(s_I\models \phi )\ge \beta .$$

$${\sigma }_{ha}(s,\alpha )=b(s)\cdot {\sigma }_h(s,\alpha )+(1-b(s))\cdot {\sigma }_a(s,\alpha ).$$

$$\sum _{\alpha \in Act}\delta (s,\alpha )=0\forall s\in S.$$

$$\delta (\sigma )(s,\alpha )=\sigma (s,\alpha )+\delta (s,\alpha )\forall s\in S,\alpha \in Act.$$

$$\text{maximize}\sum _{s\in B}{x}_{{\sigma }_{ha}}(s)$$

$$\forall s\in S_r.$$

$$\sum _{\alpha \in Act}{x}_{{\sigma }_{ha}}(s,\alpha )=\sum _{s^{\prime }\in S_r}\sum _{\alpha \in Act}\mathcal{P}(s^{\prime },\alpha ,s)x_{{\sigma }_{ha}}(s^{\prime },\alpha )+{\alpha }_s$$

$$\forall s\in B.$$

$$x_{{\sigma }_{ha}}(s)=\sum _{s^{\prime }\in S_r}\sum _{\alpha \in Act}\mathcal{P}(s^{\prime },\alpha ,s)x_{{\sigma }_{ha}}(s^{\prime },\alpha )+{\alpha }_s$$

$$\sum _{s\in B}{x}_{{\sigma }_{ha}}(s)\ge \beta$$

$${\sigma }_{ha}(s,\alpha )=\frac{x_{{\sigma }_{ha}}(s,\alpha )}{\sum _{\alpha \in Act}{x}_{{\sigma }_{ha}}(s,\alpha )}$$

$$\sum _{s\in B}{x}_{{\sigma }_{ha}}(s).$$

$$\text{minimize}\hat{\delta }$$

$$\forall s\in S_r.$$

$$\sum _{\alpha \in Act}{x}_{{\sigma }_{ha}}(s,\alpha )=\sum _{s^{\prime }\in S_r}\sum _{\alpha \in Act}\mathcal{P}(s^{\prime },\alpha ,s)x_{{\sigma }_{ha}}(s^{\prime },\alpha )+{\alpha }_s$$

$$\forall s\in B.$$

$$x_{{\sigma }_{ha}}(s)=\sum _{s^{\prime }\in S_r}\sum _{\alpha \in Act}\mathcal{P}(s^{\prime },\alpha ,s)x_{{\sigma }_{ha}}(s^{\prime },\alpha )+{\alpha }_s$$

$$\sum _{s\in B}{x}_{{\sigma }_{ha}}(s)\ge \beta$$

$$\forall s\in S_r.$$

$$|x_{ha}(s,\alpha )-\sum _{\alpha \in Act}{x}_{ha}(s,\alpha ){\sigma }_h(s,\alpha )|\le \hat{\delta }\sum _{\alpha \in Act}{x}_{ha}(s,\alpha ).$$

$$\forall s\in S_r.\alpha \in Act.{\sigma }_{ha}(s,\alpha )={\sigma }_h(s,\alpha )+\delta (s,\alpha ).$$

$$\forall s\in S_r.\alpha \in Act.\frac{x_{ha}(s,\alpha )}{\sum _{\alpha \in Act}{x}_{ha}(s,\alpha )}={\sigma }_h(s,\alpha )+\delta (s,\alpha )$$

$$\forall s\in S_r.\alpha \in Act.$$

$$x_{ha}(s,\alpha )=\sum _{\alpha \in Act}{x}_{ha}(s,\alpha )\left({\sigma }_h(s,\alpha )+\delta (s,\alpha )\right).$$

$$\forall s\in S_r.\alpha \in Act.$$

$$|x_{ha}(s,\alpha )-\sum _{\alpha \in Act}{x}_{ha}(s,\alpha ){\sigma }_h(s,\alpha )|\le$$

$$\sum _{s\in B}{x}_{{\sigma }_{ha}}(s)\ge \lambda$$

$$\sum _{s\in S_r\setminus G}\sum _{\alpha \in Act}C(s,\alpha )x_{{\sigma }_{ha}}(s,\alpha )\le \kappa$$

$$\text{minimize}\underset{s\in S,\alpha \in Act}{\max }|x_{{\sigma }_{ha}}(s,\alpha )-x_{{\sigma }_h}(s,\alpha )|$$

$$\text{minimize}||x_{{\sigma }_{ha}}-x_{{\sigma }_h}|{|}_{\infty }.$$

Full text

I Introduction

In shared control, a robot executes a task to accomplish the goals of a human operator while adhering to additional safety and performance requirements. Applications of such human-robot interaction include remotely operated semi-autonomous wheelchairs [15], robotic teleoperation [19], and human-in-the-loop unmanned aerial vehicle mission planning [11]. A human operator issues a command through an input interface, which maps the command directly to an action for the robot. The problem is that a sequence of such actions may fail to accomplish the task at hand, due to limitations of the interface or failure of the human operator in comprehending the complexity of the problem. Therefore, a so-called autonomy protocol provides assistance for the human in order to complete the task according to the given requirements.

At the heart of the shared control problem is the design of an autonomy protocol. In the literature, there are two main directions, based on either switching the control authority between human and autonomy protocol [28], or on blending their commands towards joined inputs for the robot [9, 18].

One approach to switching the authority first determines the desired goal of the human operator with high confidence, and then assists towards exactly this goal [10, 21]. In [14], switching the control authority between the human and autonomy protocol ensures satisfaction of specifications that are formally expressed in temporal logic. In general, switching of authority may cause a decrease in human’s satisfaction, who usually prefers to retain as much control as possible [20].

Blending incorporates providing an alternative command in addition to the one of the human operator. To introduce a more flexible trade-off between the human’s control authority and the level of autonomous assistance, both commands are then blended to form a joined input for the robot. A blending function determines the emphasis that is put on the autonomy protocol in the blending, that is, regulating the amount of assistance provided to the human. Switching of authority can be seen as a special case of blending, as the blending function may assign full control to the autonomy protocol or to the human. In general, putting more emphasis on the autonomy protocol in blending may lead to greater accuracy in accomplishing the task [8, 9, 23]. However, as humans prefer to retain control of the robot and may not approve if a robot issues a set of commands that is significantly different to the human’s command [19, 20]. None of the existing blending approaches provide formal correctness guarantees that go beyond statistical confidence bounds. Correctness here refers to ensuring safety and optimizing performance according to the given requirements. Our goal is to design an autonomy protocol that admits formal correctness while rendering the robot behavior as close to the human’s commands as possible, which is shown to enhance the human experience.

A human may be uncertain about which command to issue in order to accomplish a task. Moreover, a typical interface used to parse human’s commands, such as a brain-computer interface, is inherently imperfect. To capture such uncertainties and imperfections in the human’s decisions, we introduce randomness to the commands issued by humans. It may not be possible to blend two different deterministic commands. If the human’s command is “up” and the autonomy protocol’s command is “right”, we cannot blend these two commands to obtain another deterministic command. By introducing randomness to the commands of the human and the autonomy protocol, we ensure that the blending is always well-defined.

Take as an example a scenario involving a semi-autonomous wheelchair [15] whose navigation has to account for a randomly moving autonomous vacuum cleaner, see Fig. 1. The wheelchair needs to navigate to the exit of a room, and the vacuum cleaner moves in the room according to a probabilistic transition function. The task of the wheelchair is to reach the exit gate while not crashing into the vacuum cleaner. The human may not fully perceive the motion of the vacuum cleaner. Note that the human’s commands, depicted with the solid red line in Fig 1(a), may cause the wheelchair to crash into the vacuum cleaner. The autonomy protocol provides another set of commands, which is indicated by the solid red line in Fig 1(b), to carry out the task safely without crashing. However, the autonomy protocol’s commands deviate highly from the commands of the human. The two sets of commands are then blended into a new set of commands, depicted using the dashed red line in Fig 1(b). The blended commands perform the task safely while generating behavior as similar to the behavior induced by the human’s commands as possible.

We model the behavior of the robot as a Markov decision process (MDP) [27], which captures the robot’s actions inside a potentially stochastic environment. Problem formulations with MDPs typically focus on maximizing an expected reward (or, minimizing the expected cost). However, such formulations may not be sufficient to ensure safety or performance guarantees in a task that includes a human operator. Recently, it was shown that a reward structure is not sufficient to capture temporal logic constraints in general [17]. We design the autonomy protocol such that the resulting robot behavior satisfies probabilistic temporal logic specifications. Such verification problems have been extensively studied for MDPs [2] and mature tools exist for efficient verification [22, 7].

In what follows, we call a formal interpretation of a sequence of the human’s commands the human strategy, and the sequence of commands issued by the autonomy protocol the autonomy strategy. In [18], we formulated the problem of designing the autonomy protocol as a nonlinear programming problem. However, solving nonlinear programs is generally intractable [3]. Therefore, we proposed a greedy algorithm that iteratively repairs the human strategy such that the specifications are satisfied without guaranteeing optimality, based on [26]. Here, we propose an alternative approach for the blending of the two strategies. We follow the approach of repairing the strategy of the human to compute an autonomy protocol. We ensure that the resulting robot behavior induced by the repaired strategy deviates minimally from the human strategy, and satisfies safety and performance properties given in temporal logic specifications. We formally define the problem as a quasiconvex optimization problem, which can be solved efficiently by checking feasibility of a number of convex optimization problems [4].

The question remains how to obtain the human strategy in the first place. It may be unrealistic that a human can provide the strategy for an MDP that models a realistic scenario. To this end, we create a virtual simulation environment that captures the behavior of the MDP. We ask humans to participate in two case studies to collect data about typical human behavior. We use inverse reinforcement learning to get a formal interpretation as a strategy based on human’s inputs [1, 30]. We model a typical shared control scenario based on an autonomous wheelchair navigation [15] in our first case study. In our second case study, we consider an unmanned aerial vehicle mission planning scenario, where the human operator is to patrol certain regions while keeping away from enemy aerial vehicles.

In summary, the main contribution this paper is to efficiently synthesize an autonomy protocol such that the resulting blended or repaired strategy meets all given specifications while only minimally deviating from the human strategy. We present a new technique based on quasiconvex programming, which can be solved efficiently using convex optimization [4].

Organization. We introduce all formal foundations that we need in Section II. We provide an overview of the general shared control concept in Section III. We present the shared control synthesis problem and provide a solution based on convex optimization in Section IV. We indicate the applicability and scalability of our approach on experiments in Section V and draw a conclusion and critique of our approach in Section VI.

II Preliminaries

In this section, we introduce the required formal models and specifications that we use to synthesize the autonomy protocol, and we give a short example illustrating the main concepts.

II-A Markov Decision Processes

A probability distribution over a finite set $X$ is a function $\mu\colon X\rightarrow[0,1]\subseteq\mathbb{R}$ with $\sum_{x\in X}\mu(x)=\mu(X)=1$. The set $X$ of all distributions is $\mathit{Distr}(X)$.

Definition 1** (MDP).**

A Markov decision process (MDP) $\mathcal{M}{}=(S{},s_{\mathit{I}}{},\mathit{Act},\mathcal{P}{},\mathcal{AP}{},\mathcal{L}{})$ has a finite set $S$ of states, an initial state $s_{\mathit{I}}\in S$, a finite set $\mathit{Act}$ of actions, a transition probability function $\mathcal{P}\colon S\times\mathit{Act}\rightarrow\mathit{Distr}(S)$, a finite set $\mathcal{AP}$ of atomic propositions, and a labeling function $\mathcal{L}\colon S\rightarrow 2^{\mathcal{AP}}$ that labels each state $s\in S$ with a subset of atomic propositions $\mathcal{L}(s)\subseteq\mathcal{AP}$. We extend $\mathcal{L}$ to a sequence of states by $\mathcal{L}(s_{0}s_{1}\ldots)=\mathcal{L}(s_{0})\mathcal{L}(s_{1})\ldots$ for $s_{0},s_{1}\in S$.

MDPs have nondeterministic choices of actions at the states; the successors are determined probabilistically via the associated probability distribution. We assume that the MDP contains no deadlock states, that is, at every state at least one action is available. A cost function $C\colon S\times\mathit{Act}\rightarrow\mathbb{R}_{\geq 0}$ associates a cost to state-action pairs. If there is only a single action available at each state, the MDP reduces to a discrete-time Markov chain (MC). We use strategies to resolve the choices of actions in order to define a probability and expected cost measure for MDPs.

Definition 2** (Strategy).**

A memoryless and randomized strategy for an MDP $\mathcal{M}$ is a function $\sigma\colon S\rightarrow\mathit{Distr}(\mathit{Act})$. The set of all strategies over $\mathcal{M}$ is $\mathit{Str}^{\mathcal{M}}$.

A special case are deterministic strategies which are functions of the form $\sigma\colon S\rightarrow\mathit{Act}$ with $\sigma(s)\in\mathit{Act}(s)$. Resolving all the nondeterminism for an MDP $\mathcal{M}$ with a strategy $\sigma\in\mathit{Str}^{\mathcal{M}}$ yields an induced Markov chain $\mathcal{M}^{\sigma}$.

Definition 3** (Induced MC).**

For an MDP $\mathcal{M}{}=(S{},s_{\mathit{I}}{},\mathit{Act},\mathcal{P}{},\mathcal{AP}{},\mathcal{L}{})$ and strategy $\sigma\in\mathit{Str}^{\mathcal{M}}$, the MC induced by $\mathcal{M}$ and $\sigma$ is $\mathcal{M}^{\sigma}=(S,s_{\mathit{I}},\mathit{Act},\mathcal{P}^{\sigma})$, where

[TABLE]

In following, we assume that for a given MDP $\mathcal{M}$ and for any state $s$ $\in$ $S$, there exists a strategy $\sigma$ that induces a MC $\mathcal{M}^{\sigma}$ that ensures state $s$ is reachable under that strategy. Note that this is a standard assumption for MDPs, and we can remove the unreachable states by doing a graph search over the MDP $\mathcal{M}$ as a preprocessing step [2].

A finite or infinite sequence $\varrho^{\sigma}=s_{0}s_{1}s_{2}\ldots$ of states generated in $\mathcal{M}$ under a strategy $\sigma$ $\in$ $\mathit{Str}^{\mathcal{M}}$ is called a path. Given an induced MC $\mathcal{M}^{\sigma}$, starting from the initial state $s_{0}$, the state visited at step $t$ is given by a random variable $X_{t}$. The probability of reaching state $s^{\prime}$ from state $s$ in one step, denoted $\mathbb{P}(X_{t+1}=s^{\prime}|X_{t}=s)$ is equal to $\mathcal{P}^{\sigma}(s,s^{\prime})$. We can extend one-step reachability over a set of paths $\varrho^{\sigma}$, i.e., $\mathbb{P}(s_{0}s_{1}s_{2}\ldots s_{n})=\mathbb{P}(X_{n}=s_{n}|X_{n-1}=s_{n-1})\cdot\mathbb{P}(s_{0}s_{1}s_{2}\ldots s_{n-1})$. We denote the set of all paths in $\mathcal{M}$ under the strategy $\sigma$ by $Path^{\sigma}(\mathcal{M})$.

Definition 4** (Occupancy Measure).**

The occupancy measure $x_{\sigma}$ of a strategy $\sigma$ for an MDP $\mathcal{M}$ is defined as

[TABLE]

where $s_{t}$ and $\alpha_{t}$ denote the state and action in $\mathcal{M}$ at time step $t$. The occupancy measure $x_{\sigma}(s,\alpha)$ is the expected number of times to take action $\alpha$ at state $s$ under the strategy $\sigma$.

In our solution approach, we use the occupancy measure of a strategy to compute an autonomy protocol.

II-B Specifications

We use linear temporal logic (LTL) to specify a set of tasks [2]. A specification in LTL is built from a set $\mathcal{AP}$ of atomic propositions, $\texttt{true},\texttt{false}$ and the Boolean and temporal connectives $\wedge,\vee,\neg,\Rightarrow,\Leftrightarrow$, and $\square$ (always), $\mathcal{U}$ (until), $\lozenge$ (eventually), and $\bigcirc$ (next). An infinite sequence of subsets of $\mathcal{AP}$ defines an infinite word, and an LTL specification is interpreted over infinite words on $2^{\mathcal{AP}}$. If a word $w=w_{0}w_{1}w_{2}\ldots$ satisfies an LTL specification $\varphi$, we denote it by $w\models\varphi$.

Definition 5**.**

(DRA) A deterministic Rabin automaton (DRA) is a tuple $\mathcal{A}{}=(Q{},q_{\mathit{I}}{},\Sigma,\mu{},\mathit{Acc}{},\mathcal{L}{})$, with a finite set $Q$ of states, an initial state $q_{\mathit{I}}\in Q$, the alphabet $\Sigma$, the transition relation $\mu\colon Q\times\Sigma\rightarrow Q$ between states of a DRA, and the set of accepting state pairs $\mathit{Acc}\subseteq 2^{Q}\times 2^{Q}$.

A run of a DRA $\mathcal{A}$, denoted by $\gamma=q_{0}q_{1}q_{2}\ldots$, is an infinite sequence of states. For each $i\geq 0$, $q_{i+1}=\mu(s_{i},v_{i})$ for some $v_{i}\in\Sigma$. A run $\gamma$ is accepting if there exists a pair $(A,B)\in\mathit{Acc}$ and $n\geq 0$ such that, for all $m\geq n$, we have $q_{m}\notin A$ and there exists infinitely many $k$ that satisfies $q_{k}\in B.$ Given an LTL specification $\varphi$ with atomic propositions $\mathcal{AP}$, a DRA $\mathcal{A}_{\varphi}$ can be constructed with alphabet $2^{\mathcal{AP}}$ that accepts all words that satisfy the LTL specification $\varphi$ [2].

For an induced DTMC $\mathcal{M^{\sigma}}$ of an MDP and a strategy $\sigma$, a path $\varrho^{\sigma}$ $=$ $s_{0}s_{1}\ldots$ generates a word $w$ $=$ $w_{0}w_{1}\ldots$ where $w_{k}$ $=$ $\mathcal{L}(s_{k})$ for all $k\geq 0$. We denote the word that is generated by $\varrho^{\sigma}$ as $\mathcal{L}(\varrho^{\sigma})$. For an LTL specification $\varphi$, the set of words that is accepted by the DRA and satisfies the LTL specification $\varphi$ is given by $\{\varrho^{\sigma}\in Path^{\sigma}(\mathcal{M})\colon\mathcal{L}(\varrho^{\sigma})\models\varphi\}$ , and is measurable [2]. We define

[TABLE]

as the probability of satisfying the LTL specification $\varphi$ for an MDP $\mathcal{M}$ under the strategy $\sigma\in\mathit{Str}^{\mathcal{M}}$.

The synthesis problem is to find one particular strategy $\sigma$ for an MDP $\mathcal{M}$ such that given an LTL specification $\varphi$ and a threshold $\beta\in[0,1]$, the induced DTMC $\mathcal{M^{\sigma}}$ satisfies

[TABLE]

which implies that the strategy $\sigma$ satisfies the specification $\varphi$ with at least a probability of $\beta$.

We also consider expected cost properties $\varphi_{c}=\mathbb{E}_{\leq\kappa}(\lozenge G)$, that restricts the expected cost to reach the set $G\subseteq S$ of goal states by an upper bound $\kappa\in\mathbb{Q}$.

Example 1**.**

Fig. 2(a) depicts an MDP $\mathcal{M}$ with initial state $s_{0}$. In state $s_{0}$, the available actions are $a$ and $b$. Similarly for state $s_{1}$, the two available actions are $c$ and $d$. If action $a$ is selected in state $s_{0}$, the agent transitions to $s_{1}$ and $s_{3}$ with probabilities $0.6$ and $0.4$. For states $s_{2},s_{3}$ and $s_{4}$ we omit actions, because of the self loops.

For a safety specification $\phi=\mathbb{P}_{\geq 0.21}(\lozenge s_{2})$, the deterministic strategy $\sigma_{1}\in\mathit{Str}^{\mathcal{M}}$ with $\sigma_{1}(s_{0},a)=1$ and $\sigma_{1}(s_{1},c)=1$ induces a probability of $0.36$ to reach $s_{2}$. Therefore, the specification is not satisfied, see the induced MC in Fig. 2(b). Likewise, the randomized strategy $\sigma_{\textit{unif}}\in\mathit{Str}^{\mathcal{M}}$ with $\sigma_{\textit{unif}}(s_{0},a)=\sigma_{\textit{unif}}(s_{0},b)=0.5$ and $\sigma_{\textit{unif}}(s_{1},c)=\sigma_{\textit{unif}}(s_{1},d)=0.5$ violates the specification, as the probability of reaching $s_{2}$ is $0.25$. However, the deterministic strategy $\sigma_{\textit{safe}}\in\mathit{Str}^{\mathcal{M}}$ with $\sigma_{\textit{safe}}(s_{0},b)=1$ and $\sigma_{\textit{safe}}(s_{1},d)=1$ induces a probability of $0.16$, thus $\sigma_{\textit{safe}}\models\phi$.

II-C Strategy synthesis in an MDP

Given an MDP, and an LTL specification $\varphi$, we aim to synthesize a strategy that satisfies $\varphi$, or equivalently, a strategy that satisfies the condition in (2).

Definition 6**.**

(Product MDP) Let $\mathcal{M}{}=(S{},s_{\mathit{I}}{},\mathit{Act},\mathcal{P}{},\mathcal{AP}{},\mathcal{L}{})$ be an MDP and $\mathcal{A}{}=(Q{},q_{\mathit{I}}{},\Sigma,\mu{},\mathit{Acc}{},\mathcal{L}{})$ be a DRA. The product MDP is a tuple $\mathcal{M}_{p}{}=(S_{p}{},s_{\mathit{I}_{p}}{},\mathit{Act},\mathcal{P}_{p}{},\mathit{Acc}_{p}{})$ with a finite set $S_{p}=S\times Q$ of states, an initial state $s_{\mathit{I}_{p}}=(s_{\mathit{I}},q)\in S_{p}$ that satisfies $q=\delta(q_{\mathit{I}},\mathcal{L}(s_{\mathit{I}}))$, a finite set $\mathit{Act}$ of actions, a transition probability function $\mathcal{P}_{p}((s,q),a,(s^{\prime},q^{\prime}))=\mathcal{P}(s,a,s^{\prime})\;\text{if}\quad q^{\prime}=\delta(q,\mathcal{L}(s^{\prime}))$, and $\mathcal{P}_{p}((s,q),a,(s^{\prime},q^{\prime}))=0\;\text{if}\quad q^{\prime}\neq\delta(q,\mathcal{L}(s^{\prime}))$, a labeling function $\mathcal{L}_{p}((s,q))=\{q\}$, and the acceptance condition $\mathit{Acc}_{p}=\{(A_{1}^{p},B_{1}^{p}),\ldots,(A_{k}^{p},B_{k}^{p})\}$ where $A_{i}^{p}=S\times A_{i}$ and $B_{i}^{p}=S\times B_{i}$ for all $(A_{i},B_{i})\in\mathit{Acc}$ and $i=1,\ldots,k$.

Definition 7**.**

(AEC) The end component for the product MDP $\mathcal{M}_{p}$ is given by a pair $(C,D)$, where a non-empty set $C\subseteq S_{p}$ of states and a function $D\colon C\rightarrow\mathit{Act}$ is defined such that for any $s\in C$ we have

[TABLE]

and the induced directed graph is strongly connected. An accepting end component (AEC) is an end component that satisfies $C\cap A^{p}_{i}=\emptyset$ and $C\cap B^{p}_{i}\neq\emptyset$ for some $i\in{1,\ldots,k}.$

Given a product MDP $\mathcal{M}_{p}$, we modify it to $\mathcal{M}_{p}^{\prime}$ by making all states in the end components absorbing, i.e., for all states $s\in C$, $\mathcal{P}_{p}(s,\alpha,s)=1$ for all $\alpha\in\mathit{Act}$ in the modified MDP $\mathcal{M}_{p}^{\prime}$. Making all end components absorbing is commonly used in tools for model checking of LTL specifications in MDPs [2, 22, 7]. We further assume that all states in the end component are absorbing. The modification does not change the probability of satisfying an LTL specification as stated below.

Lemma 1**.**

(From [6]) In each end component of an MDP, there exists a strategy in each state $s\in C$ that reaches any other state $s^{\prime}\in C$ with a probability of 1.

A memoryless and randomized strategy for a product MDP $\mathcal{M}_{p}$ is a function $\sigma^{p}\colon S_{p}\rightarrow\mathit{Distr}(\mathit{Act})$. A memoryless strategy $\sigma^{p}$ is a finite-memory strategy $\sigma^{\prime}$ in the underlying MDP $\mathcal{M}$. Given a state $(s,q)\in S_{p}$, we consider $q$ to be a memory state and define $\sigma^{\prime}(\gamma)=\sigma^{p}(s,q)$, where the run $\gamma=q_{0}q_{1}q_{2}\ldots q_{n}$ satisfies $q_{n}=q$ and $\Sigma(q_{\mathit{I}},L(\rho))=s.$ For the MDPs given in Definition 1 and LTL specifications, memoryless strategy in the product MDP $\mathcal{M}_{p}$ are sufficient to achieve the maximal probability of satisfying the specification [2].

Some states in the product MDP $\mathcal{M}_{p}$ may be unreachable from the initial state $s_{\mathit{I}_{p}}$. These states do not affect the strategy synthesis in $\mathcal{M}_{p}$, and can be removed from $\mathcal{M}_{p}$. We assume that there is no unreachable states in the product MDP $\mathcal{M}_{p}$.

Let $\sigma^{p}\in\mathit{Str}^{\mathcal{M}_{p}}$ be a strategy for $\mathcal{M}_{p}$ and let $\sigma^{\prime}\in\sigma^{\mathcal{M}}$ be the strategy on $\mathcal{M}$ constructed from $\sigma^{p}$ through the procedure explained above. The paths of the MDP $\mathcal{M}$ under the strategy $\sigma^{\prime}$ satisfy the LTL specification $\varphi$ with a probability of at least $\beta$, i.e., $\mathbb{P}_{\mathcal{M}^{\sigma}}(s_{\mathit{I}}\models\varphi)\geq\beta$, if and only if the paths of the induced DTMC $\mathcal{M}_{p}^{\sigma}$ from the product MDP $\mathcal{M}_{p}$ under the strategy $\sigma^{p}$ reach and stay in some AECs in $\mathcal{M}_{p}$ with a probability of at least $\beta$ [2].

III Conceptual description of shared control

We now detail the general shared control concept adopted in this paper and state the formal problem. Consider the setting in Fig. 3. As inputs, we have a set of task specifications, a model $\mathcal{M}$ for the robot behavior, and a blending function $b$. The given robot task is described by certain performance and safety specifications $\varphi=\varphi_{1}\wedge\varphi_{2}\ldots\wedge\varphi_{n}$. For example, it may not be safe to take the shortest route because there may be too many obstacles in that route. In order to satisfy performance considerations, the robot should prefer to take the shortest route possible while not violating the safety specifications. We model the behavior of the robot inside a stochastic environment as an MDP $\mathcal{M}$.

In our setting, a human issues a set of commands for the robot to execute. It may be unrealistic that a human can grasp an MDP that models a realistic shared control scenario. Indeed, a human will likely have difficulties interpreting a large number of possibilities and the associated probability of paths and payoffs [13], and it may be impractical for the human to provide the human strategy to the autonomy protocol, due to the possibly large state space of the MDP. Therefore, we compute a human strategy $\sigma_{h}$ as an abstraction of a sequence of human’s commands, which we obtain using inverse reinforcement learning [1, 30].

We design an autonomy protocol that provides another strategy $\sigma_{a}$, which we call the autonomy strategy. Then, we blend the two strategies according to the blending function $b$ into the blended strategy $\sigma_{ha}$. The blending function reflects preference over the human strategy or the autonomy strategy. We ensure that the blended strategy deviates minimally from the human strategy.

At runtime, we can then blend commands of the human with commands of the autonomy strategy. The resulting “blended” commands will induce the same behavior as the blended strategy $\sigma_{ha}$, and the specifications are satisfied. Note that blending commands at runtime according to predefined blending function and autonomy protocol simply requires a linear combination of real values and is thus very efficient.

The shared control synthesis problem is then the synthesis of the repaired strategy $\sigma_{ha}$ such that it holds that $\sigma_{ha}\models\varphi$ while deviating minimally from $\sigma_{h}$. The deviation between the human strategy $\sigma_{h}$ and the repaired strategy $\sigma_{ha}$ is measured by the maximal difference between the two strategies in each state of the MDP. We state the problem that we study as follows.

Problem 1**.**

Let $\mathcal{M}$ be an MDP, $\varphi$ be an LTL specification, $\sigma_{h}$ be a human strategy, and $\beta$ be a constant. Synthesize a repaired strategy $\sigma_{ha}\in\mathit{Str}^{\mathcal{M}}$ that solves the following problem.

[TABLE]

For convenience, we will use the original MDP $\mathcal{M}$ instead of the product MDP $\mathcal{M}_{p}$ in what follows as all concepts are directly transferrable.

IV Synthesis of the autonomy protocol

In this section, we describe our approach to synthesize an autonomy protocol for the shared control synthesis problem. We start by formalizing the concepts of strategy blending and strategy repair. We then show how we can synthesize a repaired strategy that deviates minimally from the human strategy based on quasiconvex programming. We discuss how we can include additional specifications to the problem and discuss other measures for the human and the repaired strategy that induce a similar behavior.

IV-A Strategy blending

Given the human strategy $\sigma_{h}\in\mathit{Str}^{\mathcal{M}}$ and the autonomy strategy $\sigma_{a}\in\mathit{Str}^{\mathcal{M}}$, a blending function computes a weighted composition of the two strategies by favoring one or the other strategy in each state of the MDP [19, 8, 9].

Reference [9] argues that the weight of blending shows the confidence in how well the autonomy protocol can assist to perform the human’s task. Put differently, the blending function should assign a low confidence to the actions of the human if they may lead to a violation of the specifications. Recall Fig. 1 and the example in the introduction. In the cells of the gridworld where some actions may result in a collusion with the vacuum cleaner with a high probability, it makes sense to assign a higher weight to the autonomy strategy.

We pick the blending function as a state-dependent function that weighs the confidence in both the human strategy and the autonomy strategy at each state of the MDP $\mathcal{M}$ [19, 8, 9].

Definition 8** (Linear blending).**

Given the MDP $\mathcal{M}{}=(S{},s_{\mathit{I}}{},\mathit{Act},\mathcal{P}{},\mathcal{AP}{},\mathcal{L}{})$, two memoryless strategies $\sigma_{h},\sigma_{a}\in\mathit{Str}^{\mathcal{M}}$, and a blending function $b\colon S\rightarrow[0,1]$, the blended strategy $\sigma_{ha}\in\mathit{Str}^{\mathcal{M}}$ for all states $s\in S$, and actions $\alpha\in\mathit{Act}$ is

[TABLE]

For each $s\in S$, the value of $b(s)$ represents the “weight” of $\sigma_{h}$ at $s$, meaning how much emphasis the blending function puts on the human strategy at state $s$. For example, referring back to Fig. 1, the critical cells of the gridworld correspond to certain states of the MDP $\mathcal{M}$. At these states, we may assign a very low confidence in the human strategy. For instance at such a state $s\in S$, we might have $b(s)=0.1$, meaning the blended strategy in state $s$ puts more emphasis on the autonomy strategy $\sigma_{a}$.

IV-B Solution to the shared control synthesis problem

In this section, we propose an algorithm for solving the shared control synthesis problem. Our solution is based on quasiconvex programming which can be solved by checking feasibility of a number of convex optimization problems. We show that the result of the quasiconvex program is the repaired strategy as in Problem 1. The strategy satisfies the task specifications while deviating minimally from the human strategy. We use that result to compute the autonomy strategy $\sigma_{a}$ that may then be subject to blending.

IV-B1 Perturbation of strategies

As mentioned in the introduction, the blended strategy should deviate minimally from the human strategy. To measure the quantity of such a deviation, we introduce the concept of perturbation, which was used in [5]. To modify a (randomized) strategy, we employ additive perturbation by increasing or decreasing the probabilities of action choices in each state. We also ensure that for each state, the resulting strategy is a well-defined distribution over the actions.

Definition 9** (Strategy perturbation).**

Given the MDP $\mathcal{M}$ and a strategy $\sigma\in\mathit{Str}^{\mathcal{M}}$, a perturbation $\delta$ is a function $\delta\colon S\times\mathit{Act}\rightarrow[-1,1]$ with

[TABLE]

The perturbation value at state $s$ for action $\alpha$ is $\delta(s,\alpha)$. Overloading the notation, the perturbed strategy $\delta(\sigma)$ is

[TABLE]

IV-B2 Dual linear programming formulation for MDPs

In this section, we recall the LP formulation to compute a strategy that maximizes the probability of satisfying a specification $\varphi$ in an MDP [27, 12]. Let $B$ the set of states in accepting end components in $\mathcal{M}$ (or in fact within in the product MDP $\mathcal{M}_{p}$) and let $S_{r}$ be the set of all states that are not in $B$ and have nonzero probability of reaching a state $s\in B$. These sets can be computed in time polynomial in the size of $\mathcal{M}$ by doing a graph search over the MDP $\mathcal{M}$ [2]. In this section, we assume that there exists a strategy $\sigma\in\mathit{Str}^{\mathcal{M}}$ that satisfies an LTL formula with a probability of at least $\beta$, which can be verified in time polynomial by solving a linear programming problem [2].

The variables of the dual LP formulation are following:

• •

$x_{\sigma_{ha}}(s,\alpha)\in[0,\infty)$ for each state $s\in S_{r}$ and action $\alpha\in\mathit{Act}$ defines the occupancy measure of a state-action pair for the strategy $\sigma_{ha}$, i.e., the expected number of times of taking action $\alpha$ in state $s$.

• •

$x_{\sigma_{ha}}(s)\in[0,1]$ for each state $s\in B$ defines the probability of reaching a state $s\in B$ in an accepting end component.

[TABLE]

where $\alpha_{s}=1$ if $s=s_{\mathit{I}}$ and $\alpha_{s}=0$ if $s\neq s_{\mathit{I}}$. The constraints (8) and (9) ensure that the expected number of times transitioning to a state $s\in S$ is equal to the expected number of times to take action $\alpha$ that transitions to a different state $s^{\prime}\in S$. The constraint (10) ensures that the specification $\varphi$ is satisfied with a probability of at least $\beta$. We refer the reader to [27, 12] for details about the constraints in the LP.

For any optimal solution $x$ to the LP in (7)–(10),

[TABLE]

is an optimal strategy, and $x_{\sigma_{ha}}$ is the occupancy measure of $\sigma_{ha}$, see [27] and [12] for details. After finding an optimal solution to the LP in (7)–(10), we can compute the probability of satisfying a specification by

[TABLE]

IV-B3 Strategy repair using quasiconvex programming

Given the human strategy, $\sigma_{h}\in\mathit{Str}^{\mathcal{M}}$, the aim of the autonomy protocol is to compute the blended strategy, or the repaired strategy $\sigma_{ha}$ that induces a similar behavior to the human strategy while satisfying the specifications. We compute the repaired strategy by perturbing the human strategy, which is introduced in Definition 9. We show our formulation to compute the repaired strategy in the following Lemma.

Lemma 2**.**

The shared control synthesis problem can be formulated as the following nonlinear programming program with following variables:

• •

$x_{\sigma_{ha}}(s,\alpha)\in[0,\infty)$ for each state $s\in S_{r}$ and action $\alpha\in\mathit{Act}$ and $x_{\sigma_{ha}}(s)\in[0,1]$ for each state $s\in B$ as defined for the optimization problem in (7)–(10).**

• •

$\hat{\delta}\in[0,1]$ gives the maximal deviation between the human strategy $\sigma_{h}$ and the repaired strategy $\sigma_{ha}$.**

[TABLE]

Proof.

For any solution to the optimization problem above, the constraints in (13)–(15) ensure that the strategy computed by (11) satisfies the specification. We now show that by minimizing $\hat{\delta}$, we minimize the maximal deviation between the human strategy and the repaired strategy.

As in Definition 9, we perturb the human strategy $\sigma_{h}$ to the repaired strategy $\sigma_{ha}$ by

[TABLE]

Note that this constraint is not a function of the occupancy measure of $\sigma_{ha}$. By using the definition of occupancy measure in (11), we reformulate the constraint in (17) into the constraint

[TABLE]

or equivalently to the constraint

[TABLE]

Since we are interested in minimizing the maximal deviation, we assign a common variable $\hat{\delta}\in[0,1]$ for all state-action pairs in the MDP $\mathcal{M}$ to put an upper bound on the deviation by

[TABLE]

Therefore, by minimizing $\hat{\delta}$ subject to the constraints in (13)–(16) we ensure that the repaired strategy $\sigma_{ha}$ deviates minimally from the human strategy $\sigma_{h}$. ∎

The constraint in (20) is a nonlinear constraint. In fact, it is a quadratic constraint due to multiplication of $\hat{\delta}$ and $x_{ha}$. However, we show that the problem in (12)–(16) is a quasiconvex programming problem, which can be solved efficiently using bisection over $\hat{\delta}$ [4].

Lemma 3**.**

The constraint in (20) is quasiconvex, therefore the nonlinear programming problem in (12)–(16) is a quasiconvex programming problem.

Proof.

For a fixed $\hat{\delta}$, the set described by the inequality in (20) is convex, that is, the sublevel sets of the function are convex [4, Section 3.4]. Therefore, the constraint in (20) is quasiconvex and the nonlinear programming problem in (12)–(16) is a quasiconvex programming problem (QCP). ∎

We solve the QCP in (12)–(16) by employing bisection over the variable $\hat{\delta}$. We initialize a lower and upper bound of the maximal deviation between the human strategy and the repaired strategy to [math] and $1$ respectively. Then, we iteratively refine the bounds by solving a number of convex feasibility problems. A method to solve quasiconvex optimization problems is given in [4, Algorithm 4.1]. Our approach is given in Algorithm 1 based on the Algorithm 4.1 in [4]. We now state the main result of the paper.

Theorem 1**.**

The repaired strategy $\sigma_{ha}$ obtained from Algorithm 1 satisfies the task specifications and it deviates minimally from the human strategy $\sigma_{h}$, and is an optimal solution to the shared control synthesis problem.

Proof.

From a satisfying assignment to the constraints in (12)–(16), we compute a strategy that satisfies the specification using (11). Using Algorithm 1, we can compute the repaired strategy $\sigma_{ha}$ that deviates minimally from the human strategy $\sigma_{h}$ up to $\epsilon$ accuracy in $\left\lceil\log_{2}(\dfrac{1}{\epsilon})\right\rceil$ iterations. Therefore, Algorithm 1 computes an optimal strategy for the shared control synthesis problem. ∎

The strategy given by Algorithm 1 computes the minimally deviating repaired strategy $\sigma_{ha}$ that satisfies the LTL specification. In [18], we considered computing an autonomy protocol with a greedy approach. That approach requires solving possibly an unbounded number of LPs to compute a feasible strategy that is not necessarily optimal. On the other hand, using Algorithm 1, we only need to check feasibility of a number of LPs that can be determined to compute an optimal strategy. Note that we do not compute the autonomy strategy $\sigma_{a}$ with the QCP in (12)–(16) directly. After computing the repaired strategy $\sigma_{ha}$, we compute the autonomy strategy $\sigma_{a}$ according to the Definition 8.

Computationally, the most expensive step of the Algorithm 1 is checking the feasibility of the optimization problem in (13)–(16). The number of variables and constraints in the optimization problem are linear in the number of states and actions in $\mathcal{M}$, therefore, checking feasibility of the optimization problem can be done in time polynomial in the size of $\mathcal{M}$ with interior point methods [24]. Algorithm 1 terminates after $\left\lceil\log_{2}(\dfrac{1}{\epsilon})\right\rceil$ iterations, therefore we can compute an optimal strategy up to $\epsilon$ accuracy in time polynomial in the size of $\mathcal{M}$.

IV-B4 Additional specifications

The QCP in (12)–(16) computes an optimal strategy for a single LTL specification $\varphi$. Suppose that we are given a reachability specification $\varphi_{r}=\mathbb{P}_{\geq\lambda}(\lozenge T)$ with $T\in S$ in addition to the LTL specification $\varphi$. We can handle this specification by appending the constraint

[TABLE]

to the QCP in (12)–(16). The constraint in (21) ensures that the probability of reaching $T$ is greater than $\lambda$.

We handle an expected cost specification $\mathbb{E}_{\leq\kappa}(\lozenge G)$ for $G\subseteq S$, by adding the constraint

[TABLE]

to the QCP in (12)–(16). The constraint in (21) ensures that the expected cost of reaching $G$ is less than $\kappa$.

IV-B5 Additional measures

We discuss additional measures that can be used to render the behavior between the human and the autonomy protocol similar based on the occupancy measure of a strategy. Instead of minimizing the maximal deviation between the human strategy and the repaired strategy, we can also minimize the maximal difference of occupancy measures of the strategies. In this case, the difference between the human strategy and the repaired strategy will be smaller in states where the expected number of being in a state is higher, and will be higher if the state is not visited frequently. We can minimize the maximal difference of occupancy measures by adding the following objective to the constraints in (13)–(15):

[TABLE]

or, equivalently,

[TABLE]

The occupancy measure of the human strategy can be computed by finding a feasible solution to the constraints in (13)–(14) for the induced DTMC $\mathcal{M}^{\sigma_{ha}}$. We can also minimize further convex norms of the human strategy and the repaired strategy, such as 1-norm or 2-norm.

V Case study and experiments

We present two numerical examples that illustrate the efficacy of the proposed approach. In the first example, we consider a wheelchair scenario from Fig. 1. The goal in this scenario is to reach the target state while not crashing with the obstacle. In the second example, we consider an unmanned aerial vehicle (UAV) mission, where the objective is to survey certain regions while avoiding enemy agents.

We require an abstract representation of the human’s commands as a strategy to use our synthesis approach in a shared control scenario. We first discuss how such strategies may be obtained using inverse reinforcement learning and report on case study results.

V-A Experimental setting

We give an overview of the workflow of the experiments in Fig. 4. In an simulation environment, we collect sample data from the human’s commands. Based on these commands, we compute a human strategy $\sigma_{h}$ using maximum-entropy inverse reinforcement learning (MEIRL) [30]. After computing the human strategy, we synthesize the repaired strategy $\sigma_{ha}$ using the procedure in Algorithm 1. After synthesizing the repaired strategy, we compute the autonomous strategy using (5). We can further refine our representation of the human strategy by collecting more sample data from the human’s commands before blending with the autonomous strategy.

We model the wheelchair scenario inside an interactive Python environment. In the second scenario, we use the UAV simulation environment AMASE111https://github.com/afrl-rq/OpenAMASE, developed at Air Force Research Laboratory. AMASE can be used to simulate multi-UAV missions. The graphical user interfaces of AMASE allow humans to send commands to one or multiple vehicles at run time. It includes three main programs: a simulator, a data playback tool, and a scenario setup tool.

We use the model checker PRISM [22] to verify if the computed strategies satisfy the specification. We use the LP solver Gurobi [16] to check the feasibility of the LP problems that is given in Section IV. We also implemented the greedy approach for strategy repair in [18]. In this section, we refer to the procedure given by Algorithm 1 as QCP method, and the procedure from [18] as greedy method.

V-B Data collection

We asked five participants to accomplish tasks in the wheelchair scenario. The goal is moving the wheelchair to a target cell in the gridworld while never occupying the same cell as the moving obstacle. Similarly, three participants performed the surveillance task in the AMASE environment.

From the data obtained from each participant, we compute an individual randomized human strategy $\sigma_{h}$ via MEIRL. Reference [19] uses inverse reinforcement learning to reason about the human’s commands in a shared control scenario from human’s demonstrations. However, they lack formal guarantees on the robot’s execution.

In our setting, we denote each sample as one particular command of the participant, and we assume that the participant issues the command to satisfy the specification. Under this assumption, we can bound the probability of a possible deviation from the actual intent with respect to the number of samples using Hoeffding’s inequality for the resulting strategy, see [29] for details. Using these bounds, we can determine the required number of commands to get an approximation of a typical human behavior. The probability of a possible deviation from the human behavior is given by $\mathcal{O}(\exp(-n\gamma^{2}))$, where $n$ is the number of commands from the human and $\gamma$ is the upper bound on the deviation between the probability of satisfying the specification with the true human strategy and the probability obtained by the strategy that is computed by inverse reinforcement learning. For example, to ensure an upper bound $\gamma=0.05$ on the deviation of the probability of satisfying the specification with a probability of $0.99$, we require $1060$ demonstrations from the human.

We design the blending function by assigning a low weight to the human strategy at states where it yields a low probability of reaching the target set. Using this function, we create the autonomy strategy $\sigma_{a}$ and pass it (together with the blending function) back to the environment. Note that the repaired strategy $\sigma_{ha}$ satisfies the specification, by Theorem 1.

V-C Gridworld

The size of the gridworld in Fig. 1 is variable, and we generate a number of randomly moving (e.g., the vacuum cleaner) and stationary obstacles. An agent (e.g., the wheelchair) moves in the gridworld according to the commands from a human. For the gridworld scenario, we construct an MDP where the states represent the positions of the agent and the obstacles and the actions induce changes in the agent position.

The safety specification states that the agent has to reach a target cell while not crashing into an obstacle with a certain probability $\beta\in[0,1]$, formally $\mathbb{P}_{\geq\beta}(\neg\texttt{crash }\mathcal{U}\texttt{ target})$.

First, we report results for one particular participant in a gridworld scenario with a $8\times 8$ grid and one moving obstacle. The states of the MDP are generated by the Cartesian product of the states of the agent and the obstacle. The agent and the obstacle have four actions in all states, namely left, right, up and down. At each state, a transition to the chosen direction occurs with a probability of 0.7, and the agent transitions to each adjacent state in the chosen direction with a probability 0.15. If a transition to the wall occurs, the agent remains in the same state. We fix a particular strategy for the obstacle, and determine the transition probabilities between states as a product of transitioning to the next states for the agent and the obstacle. The resulting MDP has $2304$ states and $36864$ transitions. We compute the human strategy using MEIRL where the features are the components of the cost function of the human, for instance the distance to the obstacle and the goal state.

We instantiate the safety specification with $\beta=0.7$, which means the target should be reached with at least a probability of $0.7$. The human strategy $\sigma_{h}$ induces a probability of $0.546$ to satisfy the specification. That is, it does not satisfy the specification.

We compute the repaired strategy $\sigma_{ha}$ using the greedy and the QCP approach, and both strategies satisfy the specification with a probability larger than $\beta$. On the one hand, the maximum deviation between $\sigma_{h}$ and $\sigma_{ha}$ is 0.15 with the greedy approach, which implies that the strategy of the human and the autonomy protocol deviates at most 15% for all states and actions. On the other hand, the maximum deviation between $\sigma_{h}$ and $\sigma_{ha}$ is 0.03 with the QCP approach. The results show that the QCP approach computes a repaired strategy that induces a more similar strategy to the human strategy compared to the LP approach.

We give a graphical representation of the human strategy $\sigma_{h}$, repaired strategy $\sigma_{ha}$, and the autonomy strategy $\sigma_{a}$ in Fig. 5. For each strategy, we indicate the average probability of safely reaching the target with the QCP approach. Note that the probability of reaching the target depends on the current position of the obstacle. Therefore, the probability for satisfying a specification could be higher or lower than shown in Fig. 5. In Fig. 5, the probability of reaching the target increases with a darker color, and black indicates a probability of $1$ to reach the target. We observe that the human strategy induces a lower probability of reaching the target in most of the states, while for the repaired strategy, the probability of reaching target is higher in all cells. Note that the autonomy strategy induces a very high probability of reaching the target in each cell, but the autonomy strategy may be too safe and may not be similar to the human strategy.

To finally assess the scalability of our approach, consider Table I. We generated MDPs for different gridworlds with a different number of states and number of obstacles. We list the number of states in the MDP and the number of transitions. We report on the time that the synthesis process took with the greedy approach and QCP approach, which includes the time of solving the LPs in the greedy method or QCPs measured in seconds. It also includes the model checking times using PRISM for the greedy approach. To represent the optimality of the synthesis, we list the maximal deviation between the repaired strategy and the human strategy for the greedy and QCP approach (labeled as ”$\delta_{\textnormal{G}}$” and ”$\delta_{\textnormal{QCP}}$”). In all of the examples, we observe that the strategies obtained by the QCP approach yield autonomy strategies with less deviation to the human strategy while having similar computation time with the greedy approach.

V-D UAV mission planning

Similar to the gridworld scenario, we generate an MDP where states denote the position of the agent and the enemy agents in an AMASE scenario. Consider an example scenario in Fig. 6: The specification (or the mission) of the agent (blue UAV) is to keep surveilling the green regions (labeled as $w_{1},w_{2},w_{3}$) while avoiding restricted operating zones (labeled as ”$\textnormal{ROZ}_{1}$, $\textnormal{ROZ}_{2}$”) and enemy agents (purple and green UAVs). We asked the participants to visit the regions in a sequence, i.e., visiting the first region, then second, and then the third region. After visiting the third region, the task is to visit the first region again to perform the surveillance.

For example, if the last visited region is $w_{3}$, then the safety specification in this scenario is $\mathbb{P}_{\geq\beta}((\neg\texttt{crash}\enskip\wedge\enskip\neg\texttt{ROZ})\enskip\mathcal{U}\enskip\texttt{target})$, where ROZ is to visit the ROZ areas and target is visiting $w_{1}$.

We synthesize the autonomy protocol on the AMASE scenario with two enemy agents. The underlying MDP has 15625 states. We use the same blending function and same threshold $\beta=0.7$ as in the gridworld example. The features to compute the human strategy with MEIRL are given by the distance to the closest ROZ, enemy agents, and the target region.

The human strategy $\sigma_{h}$ violates the specification with a probability of 0.496. Again, we compute the repaired strategy $\sigma_{ha}$ with the greedy and the QCP approach. Both strategies satisfy the specification. On the one hand, the maximum deviation between $\sigma_{h}$ and $\sigma_{ha}$ is 0.418 with the greedy approach, which means the strategies of the human and the autonomy protocol are significantly different in some states of the MDP. On the other hand, the QCP approach yields a repaired strategy $\sigma_{ha}$ that is more similar to the human strategy $\sigma_{h}$ with a maximum deviation of 0.038. The time of the synthesis procedure with the LP approach is 481.31 seconds and the computation time with the QCP approach is 749.18 seconds, showing the trade-offs between the greedy approach and the QCP approach. We see that the greedy approach can compute a feasible solution slightly faster, however the resulting blended strategy may be less similar to the human strategy compared to the QCP approach.

To assess the effect of changing the threshold of satisfying the specification, we use a different threshold $\beta=0.9$. The greedy approach did not terminate within one hour, and could not find a repaired strategy that satisfies the specification after 45 iterations. We compute a repaired strategy $\sigma_{ha}$ using the QCP approach with a maximum deviation of 0.093. The computation time with the QCP approach is 779.81 seconds, showing that the QCP approach does not take significantly more time to compute a repaired strategy even with a higher threshold. We conclude that the greedy approach may not be able to find a feasible strategy efficiently if most of the strategies in an MDP do not satisfy the specification.

We also assess the effect of adding additional constraints to the task, i.e., surveilling the next green region within a certain time step. We synthesize different policies for different expected times until the UAV reaches the next region. We summarize the results in Table II. For each different probability thresholds (labeled as ”$\beta$”) and expected times to complete the mission (labeled as ”$\kappa$”), we report the synthesis time and the maximal deviation. The results in Table II illustrate that the maximal deviation $\delta_{\textnormal{QCP}}$ increases with increasing threshold and decreasing expected time to complete the mission. For example, with the threshold $\beta=0.9$ and expected time $\kappa=20$, the maximal deviation between the human and the repaired strategy is $0.598$, which shows that the strategies of the human and the autonomy protocol can be significantly different in some states. On the other hand, with the threshold $\beta=0.7$ and expected time $\kappa=80$, the maximal deviation between the human strategy and the repaired strategy is $0.054$, which is significantly smaller than the previous examples. We also note that there is no significant difference in synthesis time for different thresholds and expected times.

V-E Effect of changing the human strategy

In this section, we investigate how changing the human strategy changes the strategy of the autonomy protocol. We perturb the human strategy from the previous example using (6) with different perturbation functions $\delta$. We use three different values for maximal perturbation for every state and action between the human strategy and the repaired strategy and two different thresholds to satisfy the specification with $\beta=0.7$ and $\beta=0.9$.

We summarize our results in Table III. We generated three different perturbed human strategies with perturbation functions that have a different maximal perturbation (labeled as ”$\delta_{\textnormal{max}}$”). We report the maximal deviation between the autonomy protocol that is synthesized using the original human strategy and the perturbed human strategy (labeled as $\delta_{\textnormal{ap}}$), the time that the synthesis process took with the QCP approach (labeled as ”QCP synth.”), and the maximal deviation between the perturbed human strategies and the repaired strategies (labeled as $\delta_{\textnormal{QCP}}$).

The results in Table III show that the maximal deviation between the repaired strategy and the human strategy does not depend on the perturbation, and it depends on the threshold of satisfying the specification. The maximal deviation between the repaired strategies increases with larger perturbations introduced to the human strategy and with a larger threshold $\beta$. The values for $\delta_{\textnormal{QCP}}$ show that the maximal deviation between the human strategy and the repaired strategy does not depend heavily on a specific human strategy, and it mostly depends on the threshold. We also note that the synthesis time is similar for all cases.

VI Conclusion and Critique

We introduced a formal approach to synthesize an autonomy protocol in a shared control setting subject to probabilistic temporal logic specifications. The proposed approach utilizes inverse reinforcement learning to compute an abstraction of a human’s behavior as a randomized strategy in a Markov decision process. We designed an autonomy protocol such that the resulting robot strategy satisfies safety and performance specifications. We also ensured that the resulting robot behavior is as similar to the behavior induced by the human’s commands as possible. We synthesized the robot behavior using quasiconvex programming. We showed the practical usability of our approach through case studies involving autonomous wheelchair navigation and unmanned aerial vehicle planning.

There is a number of limitations and also possible extensions of the proposed approach. First of all, we computed a globally optimal strategy by bisection, which requires checking feasibility of a number of linear programming problems. A direct convex formulation of the shared control synthesis problem would make computing the globally optimal strategy more efficient.

We assumed that the human’s commands are consistent through the whole execution, i. e., the human issues each command to satisfy the specification. Also, this assumption implies the human does not consider assistance from the robot while providing commands - and in particular, the human does not adapt the strategy to the assistance. It may be possible to extend the approach to handle non-consistent commands by utilizing additional side information, such as the task specifications.

Finally, in order to generalize the proposed approach to other task domains, it is worth to explore transfer learning [25] techniques. Such techniques will allow us to handle different scenarios without requiring to relearn the human strategy from the human’s commands.