Passive-Aggressive Learning and Control

TL;DR

This paper introduces a passive-aggressive control method for scalar systems that adaptively learns system parameters while ensuring stability under adversarial disturbances, balancing exploration and exploitation.

Contribution

It presents a novel controller that simultaneously stabilizes the system and learns unknown parameters, especially under adversarial conditions, with a natural optimization-based design.

Findings

Controller guarantees global stability and improves bounds on state deviation.

Effective in stabilizing unstable, adversarial system dynamics.

Demonstrated efficiency through numerical simulations.

Abstract

In this work, we investigate the problem of simultaneously learning and controlling a system subject to adversarial choices of disturbances and system parameters. We study the problem for a scalar system with $l_\infty$-norm bounded disturbances and system parameters constrained to lie in a known bounded convex polytope. We present a controller that is globally stabilizing and gives continuously improving bounds on the worst-case state deviation. The proposed controller simultaneously learns the system parameters and controls the system. The controller emerges naturally from an optimization problem and balances exploration and exploitation in such a way that it is able to efficiently stabilize unstable and adversarial system dynamics. Specifically, if the controller is faced with large uncertainty, the initial focus is on exploration, retrieving information about the system by applying state-feedback controllers with varying gains and signs. In a pre-specified bounded region around the origin, our control strategy can be seen as passive in the sense that it learns very little information. Only once the noise and/or system parameters act in an adversarial way, leading to the state exiting the aforementioned region for more than one time-step, our proposed controller behaves aggressively in that it is guaranteed to learn enough about the system to subsequently robustly stabilize it. We end by demonstrating the efficiency of our methods via numerical simulations.

Figures

Equations

$$x_{n+1}$$

$$\left|w_n\right|\le \eta \left[\begin{array}{c}a\\ b\end{array}\right]\in {\mathcal{P}}_0$$

$$b\ne 0\forall \left[\begin{array}{c}a\\ b\end{array}\right]\in {\mathcal{P}}_0,$$

$$\left|x_{1:N}-\left[x_{0:N-1},u_{0:N-1}\right]\left[\begin{array}{c}a\\ b\end{array}\right]\right|\le 1\eta ,$$

$$\mathcal{S}(x_{0:N},u_{0:N-1})$$

$$:=$$

$$\mathcal{S}(x_{0:N},u_{0:N-1})=\bigcap _{i=0}^{N-1}\mathcal{S}(x_{i:i+1},u_i).$$

$$V\left(x_{0},\mathcal{P}_{0}\right)\begin{array}[]{rl}\\ :=\min\limits_{u_{0:\infty}}Q^{1:\infty}(x_{0},\mathcal{P}_{0},u_{0:\infty})\end{array}$$

$$Q^{1:N}(x_0,{\mathcal{P}}_0,u_{0:N-1})$$

$$:=$$

$$=$$

$${\mathcal{R}}_{\eta }^N(x_0,u_{0:N},{\mathcal{P}}_0)$$

$$=$$

$$\displaystyle V_{RSF}\left(x_{0},\mathcal{P}_{0}\right)=\begin{array}[]{rl}\min\limits_{k}&Q^{1:\infty}\left(x_{0},\mathcal{P}_{0},u_{0:\infty}\right)\\ \text{s.t.}&u_{n}=kx_{n},\forall n\in\mathbb{N}\end{array}$$

$$V(x_0,{\mathcal{P}}_0)\le V_{RSF}(x_0,{\mathcal{P}}_0).$$

$$\lambda (\mathcal{P})$$

$$K(\mathcal{P})={\text{argmin}}_k\underset{\left[\begin{array}{c}a\\ b\end{array}\right]\in \mathcal{P}}{\max }\left|a+bk\right|$$

$$V_{RSF}(x_0,{\mathcal{P}}_0)=\left(\lambda ({\mathcal{P}}_0)|x_0|+\eta \right)\vee \left(\frac{\eta }{1-\lambda ({\mathcal{P}}_0)}\right).$$

$$\left|x_n\right|\le {\gamma }_n={\left|a+bk\right|}^n\left|x_0\right|+\frac{{\left|a+bk\right|}^n-1}{\left|a+bk\right|-1}\eta$$

$${\Vert {\gamma }_{1:\infty }\Vert }_{\infty }=\left(\left|a+bk\right||x_0|+\eta \right)\vee \left(\frac{\eta }{1-\left|a+bk\right|}\right)$$

$$V_{RSF}(x_0,{\mathcal{P}}_0)$$

$$=$$

$${\gamma }_n^{RSF}=\lambda {\left({\mathcal{P}}_0\right)}^n\left|x_0\right|+\frac{\lambda {\left({\mathcal{P}}_0\right)}^n-1}{\lambda \left({\mathcal{P}}_0\right)-1}\eta$$

$${\mathcal{R}}_{\eta }^N(\alpha x_0,\left\{u_n\right\},{\mathcal{P}}_0)=\alpha {\mathcal{R}}_{\eta /\alpha }^N(x_0,\left\{1/\alpha u_n(\alpha x_{0:N})\right\},{\mathcal{P}}_0)$$

$$V(x_0,{\mathcal{P}}_0)$$

$$=$$

$$\underset{x_1\in {\mathcal{R}}_{\eta }^1(x_0,{\mathcal{P}}_0,u_0)}{\max }V\left(x_1,{\mathcal{P}}_0\cap \mathcal{S}(x_{0:1},u_0)\right)]$$

$$V\left(x_0,{\mathcal{P}}_0\right)$$

$$=$$

$$\underset{\left[\begin{array}{c}a\\ b\end{array}\right]\in {\mathcal{P}}_0\cap \mathcal{S}(x_{0:1},u_0)}{\max }\underset{{\Vert w_{1:\infty }\Vert }_{\infty }\le \eta }{\max }|x_1|\vee {\Vert x_{2:\infty }\Vert }_{\infty }]$$

$$\text{ s.t. dynamics eqrefeq:dyn},$$

$$V\left(x_0,{\mathcal{P}}_0\right)$$

$$=$$

$$\dots \underset{u_{1:\infty }}{\min }\underset{\left[\begin{array}{c}a\\ b\end{array}\right]\in {\mathcal{P}}_0\cap \mathcal{S}(x_{0:1},u_0)}{\max }\underset{{\Vert w_{1:\infty }\Vert }_{\infty }\le \eta }{\max }\Vert x_{2:\infty {\Vert }_{\infty }})]$$

$$=\underset{u_0}{\min }[Q^{1:1}(x_0,{\mathcal{P}}_0,u_0)\vee$$

$$\underset{x_1\in {\mathcal{R}}_{\eta }^1(x_0,u_0,{\mathcal{P}}_0)}{\max }V(x_1,{\mathcal{P}}_0\cap \mathcal{S}(x_{0:1},u_0),u_{1:\infty })]$$

$$u_i(x_{0:i})={\mathrm{argmin}}_u\underset{x_{i+1}\in {\mathcal{R}}_{\eta }^1(x_i,{\mathcal{P}}_i,u)}{\max }|x_{i+1}|\vee$$

Full text

I Introduction

With the proliferation of big-data, and the success of machine-learning algorithms being applied to planning and control problems, there has been a renewed interest in combining and applying learning and control to continuous systems. Modern results build on the foundational ideas of adaptive control [1, 2], which we cannot hope to adequately survey here, but place an emphasis on finite-time, rather than asymptotic, guarantees of performance and stability.

To the best of our knowledge, recent results of this nature have focused on the stochastic setting wherein system parameters are unknown, and must be identified despite stochastic excitations to the system. By combining concentration results from high-dimensional statistics with techniques from robust and optimal control, regret and performance bounds can be obtained as a function of the number of data points seen by the controller. Notable examples include [3, 4, 5, 6, 7], which provide varying degrees of guarantees and practically applicable algorithms. However, as far as we are aware, no comparable results exist for the setting of bounded but adversarial process noise and parametric uncertainty.

Recently it has been shown that in the $l_{\infty}$ bounded adversarial setting, solutions to the state-estimation problem [8, 9] and the robust control problem subject to quantization and delay in the control loop [10] admit particularly intuitive and appealing forms. This work shows that the same holds true for a joint learning and control problem.

Our main contribution is what we call a passive-aggressive learning and control algorithm that trades off between identifying the true system parameters and stabilizing the system. The defining feature of this controller is that unless the system parameters and noise act in an adversarial way, pushing the state sufficiently far away from the origin, it is content with passively observing the state evolution and updating its uncertainty set. However, when the system conspires to push the state sufficiently far from the origin, it aggressively learns the system parameters and applies control actions aimed at stabilizing the system.

The rest of the paper is organized as follows: in Section II we define the problem and the necessary notions of consistent parameter sets given a sequence of observations. In Section III we then show that in the case of “strongly stabilizable” initial parameter uncertainty sets, a simple static state-feedback policy is sufficient to guarantee robust stability for all possible choices of system realization. We then build on this result in Section IV to show that if the controller updates the set of feasible system parameters with each observation, the controller performance can be strictly improved. Finally, in Section V we consider the case of general initial uncertainty sets, and show that if a two-stage controller is applied, then the uncertainty set can eventually be reduced to one that is strongly stabilizable, allowing us to switch to the aforementioned control policies. We demonstrate the efficacy of our approach in Section VI, and end with conclusions and future work in Section VII.

II System and Problem Definition

II-A System Dynamics

We consider the scalar linear discrete-time system

[TABLE]

with the state $x_{n}$, the disturbance $w_{n}$ and the causal controllers $u_{n}(x_{0:n})$ where we use the notation $x_{0:n}$ to refer to the stacked vector $\left[x_{0},x_{1},\dots,x_{n}\right]^{\mathrm{T}}$. We assume that the disturbance $w_{n}$ is $l_{\infty}$ bounded by $\eta$, and that the state-space parameters $a$ and $b$ are unknown constants, but that are constrained to lie in a known bounded convex polytope $\mathcal{P}_{0}$.

We furthermore assume that

[TABLE]

i.e., that the system is controllable for all possible realizations of the unknown state-space parameters $(a,b)$. To simplify notation, we will refer to the controllers as $u_{n}$, where the subscript reminds that $u_{n}$ is a function of $x_{0:n}$.

II-B Consistent Sets

We denote by $x_{i:j}$ and $u_{i:j}$ the stacked vector of state values $x_{n}$ and control inputs $u_{n}$, respectively, for $i\leq n\leq j$. It follows from the dynamics (1) that at time $N$, the following entry-wise inequality must hold for the true parameters $(a,b)$.

[TABLE]

where $\mathbb{1}$ is the all ones vector of compatible dimension. This inequality therefore allows us to characterize the subset of the initial uncertainty set $\mathcal{P}_{0}$ that is consistent with the observed state and control input histories given the known bound $\eta$ on the magnitude of the disturbance process $w_{n}$.

This motivates the following definition of the consistent set at time $N$:

[TABLE]

It then follows that given state and control histories $x_{0:N}$, $u_{0:N-1}$ and the initial uncertainty set $P_{0}$, we have that $[a,b]^{T}$ lies in the bounded convex polytope $P_{0}\cap\mathcal{S}(x_{0:N},u_{0:N-1})$.

For $N=1$, the set $\mathcal{S}(x_{0:1},u_{0})$ reduces to a slice of thickness $2\eta/\sqrt{x^{2}_{0}+u^{2}_{0}}$ with normal vectors $\pm[x_{0},u_{0}]^{T}$ in parameter space (see Fig.1). This motivates the following recursive definition of the consistent set $\mathcal{S}(x_{0:N},u_{0:N-1})$ at time $N$ as the intersection of $N$ such slices:

[TABLE]

II-C Problem Statement

Our objective is to find the best causal control strategy $u_{k}(x_{0:k},\mathcal{P}_{0})$ that minimizes the worst-case state deviation $\left\|x_{1:\infty}\right\|_{\infty}$ despite adversarial noise $w_{0:\infty}$ and system parameter choices $[a,b]\in\mathcal{P}_{0}$.

Formally, we seek a solution to the following infinite-horizon min-max problem

[TABLE]

where we define $Q^{1:N}(x_{0},\mathcal{P}_{0},u_{0:{N-1}})$ as

[TABLE]

where (11) is an equivalent reformulation using the following definition.

Definition II.1**.**

Define the N-step reachable set $\mathcal{R}^{N}_{\eta}(x_{0},u_{0:\infty},\mathcal{P}_{0})$ as the set of possible trajectories $x_{1:N}$, given the initial condition $x_{0}$, the initial uncertainty set $\mathcal{P}_{0}$, the controllers $u_{0:\infty}$ and the disturbance bound $\eta$, i.e.

[TABLE]

Our approach is to begin with what we term strongly stabilizable initial uncertainty sets $\mathcal{P}_{0}$, that is to say initial uncertainty sets for which an appropriately chosen static state-feedback gain is guaranteed to be stabilizing for all realizations of the system parameters $(a,b)$. We show that such a policy is optimal for optimization problem (7) restricted to static memoryless control policies. We then show that by adding an adaptive element to such a control policy, the performance can be further improved, and that an exploration/exploitation strategy naturally emerges. Finally, we tackle the case of general initial uncertainty sets and show that after an initial “passive” learning phase, the uncertainty set is eventually “aggressively” reduced to one that is strongly stabilizable, allowing for our previously derived adaptive strategy to be applied.

III Robust Static State-Feedback for Strongly Stabilizable Uncertainty Sets

In this section, we consider a restriction of optimization problem (7) to static state-feedback control policies, i.e., we restrict $u_{n}=kx_{n}$ for all $n\geq 0$. The resulting optimization problem then reads as

[TABLE]

As we are restricting ourselves to static state-feedback polices, it follows that $V_{RSF}(x_{0},\mathcal{P}_{0})$ is an upper bound for $V(x_{0},\mathcal{P}_{0})$, i.e.:

[TABLE]

We consider this simpler problem as it has several appealing properties. First, $V_{RSF}\left(x_{0},\mathcal{P}_{0}\right)$ and the corresponding minimizing $k^{*}$ can be solved for in closed form. Further it motivates the definition of strongly stabilizable initial uncertainty sets $\mathcal{P}_{0}$, which naturally captures how difficult an uncertain system can be to stabilize. To that end, we introduce the following measure of stabilizability for an uncertainty set $\mathcal{P}$.

Definition III.1**.**

We use $\lambda(\mathcal{P})$ to denote the stability margin of the parameter set $\mathcal{P}$, and define it as

[TABLE]

Furthermore, we call the corresponding minimizer

[TABLE]

the gain of the parameter set $\mathcal{P}$.

The stability margin of a set $\lambda(\mathcal{P})$ is a functional mapping sets to $\mathbb{R}^{+}_{0}$ and describes the smallest system eigenvalue achievable by a constant state-feedback, assuming worst case parameter choice of $\left[a,b\right]^{T}\in\mathcal{P}$. As presented in the following Lemma, it follows that if the initial uncertainty set $\mathcal{P}_{0}$ satisfies $\lambda(\mathcal{P}_{0})<1$, then we can use $k=K(\mathcal{P}_{0})$ as a state-feedback control law to stabilize the system for all parameters in $\mathcal{P}_{0}$. We will refer to such initial uncertainty sets $\mathcal{P}_{0}$ as strongly stabilizable.

In the following we let $a\vee b:=\max\left\{a,b\right\}$ to help simplify notation.

Lemma III.1** (Comparison-Lemma).**

Let $v_{n}$ be a non-negative sequence in $\mathbb{R}$ that satisfies $v_{n+1}\leq f_{n}(v_{n})$ with non-decreasing functions $f_{n}$, then $v_{n}$ is bounded above by the sequence $\gamma_{n}$, where $\gamma_{0}=v_{0}$ and $\gamma_{n+1}=f_{n}(\gamma_{n})$.

Proof.

The result follows by induction from $\gamma_{0}\leq v_{0}$ and $v_{n+1}\leq f_{n}(v_{n})\leq f_{n}(\gamma_{n})=\gamma_{n+1}$. ∎

Lemma III.2**.**

If the initial uncertainty set $\mathcal{P}_{0}$ is strongly stabilizable (i.e., if $\lambda(\mathcal{P}_{0})<1$), then the problem (18) attains its optimum with the controller $u^{RSF}_{n}=K(\mathcal{P}_{0})x_{n}$ and

[TABLE]

Conversely, if the initial uncertainty set $\mathcal{P}_{0}$ is not strongly stabilizable (i.e., if $\lambda(\mathcal{P}_{0})\geq 1$), then for any choice of $k$, it holds that $V_{RSF}(x_{0},\mathcal{P}_{0})=\infty$.

Proof.

Fix $u_{n}=kx_{n}$ for some $k$ and consider the case $\lambda(\mathcal{P}_{0})\geq 1$. By definition (III.1), $\exists\left[a^{*},b^{*}\right]^{\mathrm{T}}\in\mathcal{P}_{0}$ such that $\left|a^{*}+b^{*}k\right|\geq 1$ and it is easy to see that $\forall x_{0}\in\mathbb{R}$ we can find $w_{0:\infty}$ to make $x_{n}$ grow unbounded. Therefore, we conclude $V_{RSF}(x_{0},\mathcal{P}_{0})=\infty$ if $\lambda(\mathcal{P}_{0})\geq 1$.

For the second part of the proof, notice that $\left|x_{n+1}\right|\leq\left|(a+bk)\right|\left|x_{n}\right|+\eta$ and by the comparison lemma (III.1), we obtain the bound

[TABLE]

Now notice that $\gamma_{n}$ is monotonic and if $\lambda(\mathcal{P}_{0})<1$ we have that $\left\|\gamma_{1:\infty}\right\|_{\infty}$ is bounded and it takes either the value $|\gamma_{1}|$ or $\lim_{n\rightarrow\infty}|\gamma_{n}|$. Therefore

[TABLE]

Furthermore, notice that for each $n$ and fix $a,b$ and $k$, we can construct a sequence $w_{0:\infty}$ such that $|x_{n}|=\gamma_{n}$. Hence, we can rewrite the optimization problem (18) equivalently as

[TABLE]

We can now conclude the desired statement (22), by noticing that expression (24) is monotonic in $|a+bk|$ and therefore we obtain the optimum value at $k^{*}=K(\mathcal{P}_{0})$. ∎

Following the arguments of the proof of Lemma (III.2), we also obtain the following performance bound of the robust state-feedback (RSF) controller $u_{n}=K(\mathcal{P}_{0})x_{n}$:

Corollary III.1** (RSF bound).**

Assume the dynamics (1) with $\lambda\left(\mathcal{P}_{0}\right)<1$ and $u_{n}=K\left(\mathcal{P}_{0}\right)x_{n}$, then $\left|x_{n}\right|\leq\gamma^{RSF}_{n}$, where

[TABLE]

Although the RSF controller is guaranteed to stabilize systems with strongly stabilizable uncertainty sets $\mathcal{P}_{0}$ and (22) provides a finite upper bound to $V(x_{0},\mathcal{P}_{0})$, it does so in an inefficient way. In particular, it does not update its control policy to reflect the fact that with each observation, more information about the underlying true parameters is revealed. As previously discussed, the observations $x_{0:N},u_{0:N-1}$ allow us to reduce the space of consistent parameters $[a,b]^{T}$ to be $P_{0}\cap S(x_{0:N},u_{0:N-1})$. In what follows we improve upon the static state-feedback policy results of this section, and ultimately show that learning is necessary to compute stabilizing controllers for general initial uncertainty sets $\mathcal{P}_{0}$.

IV Robust Adaptive State-Feedback for Strongly Stabilizable Uncertainty Sets

Keeping our focus on strongly stabilizable initial uncertainty sets, we propose two controllers that strictly outperform the static state-feedback policy $k=K(\mathcal{P}_{0})$ defined in the previous section. The following adaptive schemes, which we call weakly adaptive RSF (WRSF) and strongly adaptive RSF (SRSF), simultaneously learn the system dynamics while controlling the system. The latter algorithm decides at every time-step $n$ between a control action that reduces $\left|x_{n+1}\right|$ and an exploratory control action that leads to more information about the system parameters. Our key result is a decomposition theorem that exploits the fact that the control policy at time $n$ is allowed to be a function of all past state-measurements $x_{0:n}$.

IV-A Decomposition and Properties of $V(x_{0},\mathcal{P}_{0})$

Using the definition (11) and Lemma (IV.1), it is easy to derive the following properties of $V(x_{0},\mathcal{P}_{0})$:

Lemma IV.1**.**

$\forall\alpha>0$* holds*

[TABLE]

Proof.

This follows by change of variables and using definition (II.1). ∎

Corollary IV.1**.**

$\forall x_{0}$* holds $V(x_{0},\mathcal{P}_{0})=V(-x_{0},\mathcal{P}_{0})$*

We then have that the following decomposition theorem holds:

Theorem IV.1**.**

For the cost-to-go function $V$ as defined in optimization problem (7), it holds that

[TABLE]

Proof.

First notice that we can write

[TABLE]

where the last equality follows from the fact that the state $x_{1}$ is known to the sequence of control actions $u_{1:\infty}$, and that $\|x_{1:\infty}\|_{\infty}=|x_{1}|\vee\|x_{2:\infty}\|_{\infty}$. This last line can further be rewritten as

[TABLE]

where the first equality follows from the fact that $\max_{x}f(x)\vee g(x)=(\max_{x}f(x))\vee(\max_{x}g(x))$, and the second from the definition of the cost-to-go function $V$, as defined in (7), and from the identity $\max\limits_{x_{1}\in\mathcal{R}^{1}_{\eta}(x_{0},u_{0},\mathcal{P}_{0})}|x_{1}|=Q^{1:1}(x_{0},\mathcal{P}_{0},u_{0}(x_{0}))$. ∎

Theorem IV.1 sheds light on the structure of the optimal control policy. Specifically, let $\mathcal{P}_{i}:=\mathcal{P}_{0}\cap\mathcal{S}(x_{0:i},u_{0:i-1})$; then the optimal control action at time $i$ is given by111This follows from Theorem IV.1 by a simple induction argument which is omitted in the interest of space.

[TABLE]

In particular, we see that the control action $u_{i}$ is a function of both the state history $x_{0:i}$ and the updated uncertainty set $\mathcal{P}_{i}$, and naturally results in a trade-off between exploration and exploitation. If the first term dominates the cost function, this indicates that the controller is in an exploitation mode, using its gathered information on the uncertainty set to minimize state deviation. In contrast, if the second term dominates, this can be interpreted as an exploration action aimed at reducing the effects of parametric uncertainty on future state deviations.

IV-B Weakly and Strongly Adaptive Robust State-Feedback Controller

Unfortunately, Eq. (27) does not provide a practical means of computing an optimal controller. Nevertheless, we can approximate (27) by using $V_{RSF}$ as an upper bound for the cost-to-go function in (27). In particular, we will define the weakly adaptive robust state-feedback (WRSF) controller as the solution to the optimization problems

[TABLE]

and the strongly adaptive robust state-feedback (SRSF) controller as the solution to the optimization problems

[TABLE]

Accordingly we will define $V_{WRSF}(x_{n},\mathcal{P}_{n})$ and $V_{SRSF}(x_{n},\mathcal{P}_{n})$ as

[TABLE]

After some standard manipulation, we can formulate the resulting controllers as

[TABLE]

$u^{WRSF}_{n}$ turns out to present a simple yet significantly better strategy than the RSF controller, as it is basically using the information of recent observations to recompute an RSF controller. Therefore it is not surprising that we have $V_{WRSF}(x_{n},\mathcal{P}_{n})=V_{RSF}(x_{n},\mathcal{P}_{n})$

To apply $u^{SRSF}$ on the other hand, requires the solution of a scalar min-max problem at every time-step. Notice from (36), how the objective of SRSF policy can be seen as a exploration vs. exploitation trade-off, as it incorporates the reduced uncertainty of the next time step. As we will show in the following theorem, if the initial uncertainty set is strongly stabilizable, then this SRSF policy is stabilizing and performs at least as well as the WRSF and RSF policy.

Theorem IV.2** (SRSF vs. WRSF vs. RSF).**

Let $x^{RSF}_{n}$, $x^{WRSF}_{n}$, $x^{SRSF}_{n}$ be sequences generated from running $u^{RSF}$, $u^{WRSF}$ and $u^{SRSF}$ in closed loop with the same initial condition $x_{0}$ and assume a strongly stabilizable $\mathcal{P}_{0}$ with the usual dynamics (1). Furthermore, assume that all sequences obtain the same uncertainty sets $\mathcal{P}_{n}$, then $x^{RSF}_{n}$, $x^{SRSF}_{n}$ and $x^{WRSF}_{n}$ are upper bounded by

[TABLE]

where

[TABLE]

with $\gamma^{SRSF}_{0}=\gamma^{WRSF}_{0}=\gamma^{RSF}_{0}=|x_{0}|$. Furthermore, we have $\gamma^{SRSF}_{n}\leq\gamma^{WRSF}_{n}\leq\gamma^{RSF}_{n}$

Proof.

While we have proven the first part already for the RSF controller, we can obtain the performance bounds for WRSF and SRSF, by observing from their definition that they are upperbounds on the worst-case adversarial dynamics. Therefore, at every-time step $n$ holds

[TABLE]

Then using the comparison lemma (III.1), we can establish $|x^{SRSF}_{n}|\leq\gamma^{SRSF}_{n}$ and $|x^{WRSF}_{n}|\leq\gamma^{WRSF}_{n}$. The inequality $\gamma^{SRSF}_{n}\leq\gamma^{WRSF}_{n}\leq\gamma^{RSF}_{n}$ follows, by observing that

[TABLE]

and noticing that $V_{WRSF}$ and $V_{RSF}$ are monotonic increasing in $|x|$ and by using comparison Lemma (III.1). ∎

V Passive Aggressive Feedback Controller

In this section we introduce a control policy that is applicable to initial uncertainty sets that are not strongly stabilizable. The controller evolves according to two stages: at first, a “passive-aggressive” feedback controller is deployed that is used as long as the consistent set $\mathcal{P}_{0}\cap\mathcal{S}(x_{0:N},u_{0:N-1})$ is not strongly stabilizable. We show that this control policy is guaranteed to shrink the initial uncertainty set to one that is strongly stabilizable once the state becomes sufficiently large. Once the uncertainty set has been reduced to a strongly stabilizable one, the controller switches to the ARSF strategy described in the previous section and drives the state to the origin.

Definition V.1**.**

Let $\mathcal{P}_{n}$ be the remaining uncertainty in the system parameters after observing $x_{0:n}$ and $u_{0:n-1}$. Specifically

[TABLE]

where we set $\mathcal{P}_{0}$ to be the initial uncertainty set.

Definition V.2**.**

Define $k_{max}(\mathcal{P}_{n})$ as the maximum deadbeat controller gain among the parameters in $\mathcal{P}_{n}$:

[TABLE]

We begin with an intermediate result that shows if the system state is sufficiently large, then the stability margin of the uncertainty set can be reduced by an amount governed by the noise bound $\eta$ and the size of the state itself. In this way, there is a notion of signal-to-noise that comes into play in the ability to learn an uncertainty set.

Theorem V.3** (Passive-Aggressive Learning).**

Let $\mathcal{P}_{0}$ be the initial (not necessarily strongly stabilizable) uncertainty set and fix positive constants $\lambda^{*},p>0$ satisfying $\lambda^{*}>\frac{1}{p}$. Consider the system (1) with the following time-varying state-feedback controller:

[TABLE]

where

[TABLE]

Then $\forall n$ that satisfy $\min\{|x_{n-1}|,|x_{n-2}|\}\geq p\eta$ holds $\lambda(\mathcal{P}_{n})\leq\lambda^{*}$.

Proof.

Assume $n_{0}$ to be such that $\min\{|x_{n_{0}-1}|,|x_{n_{0}-2}|\}\geq p\eta$ and $n_{-1}$, $n_{-2}$ to be $n_{0}-1$ and $n_{0}-2$ respectively. Then,

[TABLE]

where $\mathcal{B}(\dots)$ represents the smallest outer-bounding box set of the $\mathcal{S}(x_{n_{-2}:n_{0}},u_{n_{-2}:n_{-1}})$. Furthermore, $\Delta_{b}\mathcal{B}$ and $\Delta_{a}\mathcal{B}$ are the maximum uncertainty of parameters $a$ and $b$ in the set $\mathcal{B}$ as discussed in the appendix (A). Finally we note that the last inequality follows from the discussion on stability margins of boxed uncertainties in (A). Now, as $k_{n_{-2}}$ and $k_{n_{-1}}$ have opposite sign by construction, we obtain from app.(A):

[TABLE]

Now, since by assumption we have that $\min\{|x_{n_{-1}}|,|x_{n_{-2}}|\}\geq p\eta$, we can upper bound equations (49), (50) by

[TABLE]

Furthermore, notice that $k_{max}(\mathcal{P}_{n_{-2}})\geq k_{max}(\mathcal{P}_{n_{-1}})$ so we have

[TABLE]

which lets us further upper-bound (51) by

[TABLE]

Finally, plugging the bounds (55) and (52) into equation (48) gives us the desired result:

[TABLE]

∎

With this ability to learn the uncertainty set, we now show how a two-stage controller can lead to a stabilizing (in the BIBO sense) adaptive controller.

Theorem V.4** (Passive-Aggressive Learning and Control).**

Let $\mathcal{P}_{0}$ be an initial (not necessarily strongly stabilizable) uncertainty set, and fix positive constants $\lambda^{*},p>0$, s.t. $1-\frac{1}{p}>\lambda^{*}>\frac{1}{p}$. Consider the system (1) with the following switched control strategy:

[TABLE]

where $u^{PAL}_{n}$ represents the Passive-Aggressive Learner from (V.3) and $u^{ARSF}_{n}$ is one of the Adaptive Robust State-Feedback controller from section (IV). Then, it holds:

1. (i)

The closed loop system response $\left\|x\right\|_{\infty}$ is bounded as:

[TABLE] 2. (ii)

If $\lambda(\mathcal{P}_{n^{\prime}})\leq\lambda^{*}$, then $\forall n\geq n^{\prime}$, $x_{n}$ satisfies the convergence bounds associated with $u_{ARSF}$ from section (IV). In particular, it always satisfies the bound

[TABLE]

Proof.

(ii) follows directly from our analysis in Sec.(IV): Since $\lambda(\mathcal{P}_{n^{\prime}})\leq\lambda^{*}<1-1/p<1$, the parameter uncertainty set is strongly stabilizable and will stay so for all $n\geq n^{\prime}$. Therefore, the convergence bounds from Sec.(IV) apply.

To establish (i), we have to consider different worst-case scenarios. First, assume $\lambda(\mathcal{P}_{0})\leq\lambda^{*}$ holds, then we get the bound $\left\|x\right\|_{\infty}\leq|x_{0}|\vee p\eta$ from part (ii). On the other hand, if $\lambda(\mathcal{P}_{0})>\lambda^{*}$, then the system is being controlled by $u^{PAL}$. Recall that under that regime we have guaranteed that if $x_{n^{\prime}-1},x_{n^{\prime}-2}\leq p\eta$, we obtain a strongly stabilizable uncertainty set for $n^{\prime}$, i.e. $\lambda(\mathcal{P}_{n^{\prime}})\leq\lambda^{*}$. Therefore, for all times after $n^{\prime}$ we will apply $u^{ARSF}$ and by our previous discussion in (ii), we obtain the bound $\left\|x_{n^{\prime}:\infty}\right\|_{\infty}\leq|x_{n^{\prime}}|\vee p\eta$. It is therefore, left to consider the worst case $\left\|x_{0:n^{\prime}}\right\|_{\infty}$, where $n^{\prime}-2$ is the first time that $x$ grows outside the interval $\left[-p\eta,+p\eta\right]$. It is easy to see that we can bound the worst case transient for $\left\|x_{0:n^{\prime}}\right\|_{\infty}$, by considering the two time-step worst-case transient starting from an initial condition $x_{0}\geq p\eta$ and no prior knowledge of the system. Stating that transient bound in terms of the definitions in Sec. (II-C), we get

[TABLE]

∎

The “passive-aggressive” nomenclature is chosen because the policy is such that if the system parameters and process noise are stabilizing (i.e., keep the state close to the origin), then we only focus on learning the uncertainty set via the learning control policy (47) – this is the passive phase of the control policy. In particular, if for the specific process noise and system parameter realizations no such $n_{0}$ exists, then by definition we are guaranteed to have $|x_{n}|\leq C\eta\vee|x_{0}|$ for all $n$, where $C$ is a constant depending only on $p$, $\lambda^{*}$ and $k_{max}(\mathcal{P}_{n})$ – this follows by noting from (IV.1) that $Q^{1:2}\left(v,\mathcal{P}_{0},u^{PAL}_{0:1}\right)$ grows sublinear in $|v|$. In contrast, when the noise is such that the state is pushed sufficiently far from the origin, we are able to aggressively decrease the stability margin of the uncertainty set and switch to an ARSF policy. The result is a stabilizing control scheme that is applicable to arbitrary initial uncertainty sets $\mathcal{P}_{0}$.

VI Simulation Results

In the following we show how the passive-aggressive controller performs under different scenarios. Recall, that our plant is modeled as

[TABLE]

We set the initial uncertainty set to be

[TABLE]

and we pick the true parameters of the system to be

[TABLE]

The $u^{SRSF}$ controller is parametrized with $p=10$ and $\lambda^{*}=0.5$. Notice that this initial uncertainty set is not strongly stabilizable. In what follows, we apply the controller described in Theorem V.4 for different noise and system parameter realizations: fixed system parameters and adversarial/random noise, adversarial system parameters and noise, and fixed parameters with no noise.

Each of the figures (2)-(4) show sequences of the state $x_{n}$ and control action $u_{n}$ and the maximum and minimum feasible $a$ and $b$ of the current polytope $\mathcal{P}_{n}$. The right subfigures overlay the area of all polytopes $\mathcal{P}_{n}$ and display how the uncertainty polytopes $\mathcal{P}_{n}$ shrink with each iteration. The shade of the polytopes becomes lighter with increasing $n$.

For the simulations with adversarial noise, we choose $w_{k}=\mathrm{sign}(ax_{k}+bu_{k})$ which is easily seen to be the solution to the inner maximization problem in equation (IV.1) with the surrogate function $V_{RSF}$ to determine the adversarial noise and system parameters.

Notice that $\lambda(\mathcal{P}_{n})$ decreases monotonically, and in presence of noise the controller learns to stabilize the system within two time-steps. It is also worth noticing that in presence of no noise, the controller still chooses to perturb the system on purpose to gather information, i.e. an exploration phase naturally emerges to better identify the system parameters before a robustly stabilizing control policy is applied.

VII Conclusions and Future Work

In this paper we defined and analyzed the passive-aggressive learning and control strategy for scalar systems with bounded but adversarial process noise and parametric uncertainty. We showed that for strongly stabilizable initial uncertainty sets, sharp bounds on the state-deviation can be obtained using an ARSF control policy. We then extended these results to the general setting by proposing a two-stage controller: the first stage seeks to passively learn the system so long as the state remains sufficiently close to the origin. However, if the process and system noise are such that the state is pushed sufficiently far from the origin, the controller is able to aggressively reduce the uncertainty set to one that is strongly stabilizable, thus allowing for either the weakly or strongly ARSF policies to be applied. Future work will look to actively inject noise into the passive stage of the aforementioned two-stage control policy to expedite the learning process, as well as characterize sharp regret bounds on the proposed policy. Of additional interest is the extension of the proposed methods to the vector valued setting.

Appendix A Stability Margin Bounds

A-A Box-shaped Uncertainty Sets

Lemma A.1**.**

Let $\mathcal{B}$ be a controllable boxed uncertainty

[TABLE]

and define $\Delta_{b}\mathcal{B}$, $\Delta_{a}\mathcal{B}$, $a_{av}$, $b_{av}$, $k_{av}$ as

[TABLE]

where $a_{av}$, $b_{av}$ is the average system of the box and $k_{av}$ the corresponding deadbeat feedback. Then the stability margin of $\mathcal{B}$ can be computed as

[TABLE]

Proof.

The proof is omitted but follows by solving the following optimization problem

[TABLE]

∎

A-B Approximation for $S(x_{i:i+2},u_{i:i+1})$ in Passive-Aggressive Learning

Consider applying $u_{1}=k_{1}x_{1}$ and $u_{2}=-k_{2}x_{2}$ as a feedback controller with $k_{1}>0$, $k_{2}>0$. Then the resulting uncertainty set $S(x_{i:i+2},u_{i:i+1})$ resembles a parallelogram as shown in Fig.(6). An approximation of the stability margin $\lambda(S(x_{i:i+2},u_{i:i+1}))$ is the stability margin of its outer-bounding box, i.e. $\lambda(\mathcal{B}(x_{i:i+2},u_{i:i+1}))$. Using the notation in Fig.(6) and Lem.(A.1), the approximation can be computed as

[TABLE]

From simple geometry, notice that the shaded area $A$ can be computed in three ways:

[TABLE]

We can use these equations to solve for $x$ and $y$ and finally obtain:

[TABLE]