A Control Lyapunov Perspective on Episodic Learning via Projection to State Stability

TL;DR

This paper introduces Projection to State Stability (PSS), a new concept for analyzing the robustness of control Lyapunov functions in learned control systems, enabling better handling of uncertainties in episodic learning.

Contribution

It proposes PSS as a novel framework to analyze and bound uncertainties in control Lyapunov functions derived from learned data, enhancing robust control synthesis.

Findings

PSS effectively characterizes uncertainty in projected dynamics.

Episodic learning combined with PSS improves robustness in control.

Bounded uncertainty leads to more reliable control strategies.

Abstract

The goal of this paper is to understand the impact of learning on control synthesis from a Lyapunov function perspective. In particular, rather than consider uncertainties in the full system dynamics, we employ Control Lyapunov Functions (CLFs) as low-dimensional projections. To understand and characterize the uncertainty that these projected dynamics introduce in the system, we introduce a new notion: Projection to State Stability (PSS). PSS can be viewed as a variant of Input to State Stability defined on projected dynamics, and enables characterizing robustness of a CLF with respect to the data used to learn system uncertainties. We use PSS to bound uncertainty in affine control, and demonstrate that a practical episodic learning approach can use PSS to characterize uncertainty in the CLF for robust control synthesis.

Figures

Equations

$$\dot{\mathbf{x}}=\mathbf{f}(\mathbf{x},\mathbf{u}),$$

$$\underline{\alpha }(\Vert \mathbf{x}\Vert )\le V(\mathbf{x})$$

$$\underset{\mathbf{u}\in \mathcal{U}}{\inf }\dot{V}(\mathbf{x},\mathbf{u})$$

$$\dot{\mathbf{x}}=\mathbf{f}(\mathbf{x},\mathbf{u},\mathbf{d}),$$

$$\Vert \mathbf{x}(t)\Vert \le \beta (\Vert \mathbf{x}(0)\Vert ,t)+\gamma \left(\underset{\tau \ge 0}{\sup }\Vert \mathbf{d}(\tau )\Vert \right),$$

$$\underline{\alpha }(\Vert \mathbf{x}\Vert )\le V(\mathbf{x})$$

$$\Vert \mathbf{x}\Vert \ge \rho (\Vert \mathbf{d}\Vert )⟹\underset{\mathbf{u}\in \mathcal{U}}{\inf }\dot{V}(\mathbf{x},\mathbf{u},\mathbf{d})$$

$$\underline{\sigma }(\Vert \mathbf{x}\Vert )\le \Vert \mathbf{\Pi }(\mathbf{x})\Vert \le \overline{\sigma }(\Vert \mathbf{x}\Vert ),$$

$$\dot{\mathbf{y}}={\mathbf{D}}_{\mathbf{\Pi }}(\mathbf{x})\mathbf{f}(\mathbf{x},\mathbf{u},0)+\underset{\mathbf{\delta }}{\underbrace{{\mathbf{D}}_{\mathbf{\Pi }}(\mathbf{x})(\mathbf{f}(\mathbf{x},\mathbf{u},\mathbf{d})-\mathbf{f}(\mathbf{x},\mathbf{u},0))}},$$

$$\Vert \mathbf{x}(t)\Vert \le \beta (\Vert \mathbf{x}(0)\Vert ,t)+\gamma \left(\underset{\tau \ge 0}{\sup }\Vert \mathbf{\delta }(\tau )\Vert \right),$$

$$\mathbf{f}(\mathbf{x},\mathbf{u},\mathbf{d})=\mathbf{f}(\mathbf{x},\mathbf{u},0)+\mathbf{d},$$

$$\underline{\alpha }(\Vert \mathbf{\Pi }(\mathbf{x})\Vert )\le W(\mathbf{\Pi }(\mathbf{x}))$$

$$\Vert \mathbf{\Pi }(\mathbf{x})\Vert \ge \rho (\Vert \mathbf{\delta }\Vert )⟹\underset{\mathbf{u}\in \mathcal{U}}{\inf }\dot{W}(\mathbf{x},\mathbf{u},\mathbf{\delta })$$

$$W(\mathbf{\Pi }(\mathbf{x}))\ge \overline{\alpha }\circ \rho (\Vert \mathbf{\delta }\Vert )$$

$$⟹\underset{\mathbf{u}\in \mathcal{U}}{\inf }\dot{W}(\mathbf{x},\mathbf{u},\mathbf{\delta })\le -\alpha \circ {\overline{\alpha }}^{-1}(W(\mathbf{\Pi }(\mathbf{x}))).$$

$$\Vert \mathbf{\Pi }(\mathbf{x}(t))\Vert \le \beta (\Vert \mathbf{\Pi }(\mathbf{x}(0))\Vert ,t)+\gamma \left(\underset{\tau \ge 0}{\sup }\Vert \mathbf{\delta }(\tau )\Vert \right),$$

$$\Vert \mathbf{x}(t)\Vert \le {\underline{\sigma }}^{-1}\left(\beta (\overline{\sigma }(\Vert \mathbf{x}(0)\Vert ),t)+\gamma \left(\underset{\tau \ge 0}{\sup }\Vert \mathbf{\delta }(\tau )\Vert \right)\right).$$

$${\beta }^{\prime }(r,s)$$

$${\gamma }^{\prime }(r)$$

$$\Vert \mathbf{x}(t)\Vert \le {\beta }^{\prime }(\Vert \mathbf{x}(0)\Vert ,t)+{\gamma }^{\prime }\left(\underset{\tau \ge 0}{\sup }\Vert \mathbf{\delta }(\tau )\Vert \right).$$

$$\delta =\nabla V(\mathbf{x}{)}^{\top }(\mathbf{f}(\mathbf{x},\mathbf{u},\mathbf{d})-\mathbf{f}(\mathbf{x},\mathbf{u},0)).$$

$$\dot{V}(\mathbf{x},\mathbf{u},\delta )=\nabla V(\mathbf{x}{)}^{\top }\mathbf{f}(\mathbf{x},\mathbf{u},0)+\delta ,$$

$$\dot{V}(\mathbf{x},\mathbf{k}(\mathbf{x}),0)\le -\alpha (\Vert \mathbf{x}\Vert ),$$

$$\dot{V}(\mathbf{x},\mathbf{k}(\mathbf{x}),\delta )$$

$$\le -{\alpha }_p(\Vert \mathbf{x}\Vert )-{\alpha }_q(\Vert \mathbf{x}\Vert )+|\delta |.$$

$$\Vert \mathbf{x}\Vert \ge {\alpha }_q^{-1}(|\delta |)⟹\dot{V}(\mathbf{x},\mathbf{k}(\mathbf{x}),\delta )\le -{\alpha }_p(\Vert \mathbf{x}\Vert ).$$

$$V(\mathbf{x})\ge \overline{\alpha }\circ {\alpha }_q^{-1}(|\delta |)$$

$$⟹\dot{V}(\mathbf{x},\mathbf{k}(\mathbf{x}),\delta )\le -{\alpha }_p\circ {\overline{\alpha }}^{-1}(V(\mathbf{x})),$$

$$\dot{\mathbf{x}}=\mathbf{f}(\mathbf{x})+\mathbf{g}(\mathbf{x})\mathbf{u},$$

$$\dot{\mathbf{x}}=\hat{\mathbf{f}}(\mathbf{x})+\hat{\mathbf{g}}(\mathbf{x})\mathbf{u},$$

$$\dot{\mathbf{x}}=\hat{\mathbf{f}}(\mathbf{x})+\hat{\mathbf{g}}(\mathbf{x})\mathbf{u}+\stackrel{\mathbf{d}}{\overbrace{(\underset{\mathbf{A}(\mathbf{x})}{\underbrace{\mathbf{g}(\mathbf{x})-\hat{\mathbf{g}}(\mathbf{x})}})\mathbf{u}+\underset{\mathbf{b}(\mathbf{x})}{\underbrace{\mathbf{f}(\mathbf{x})-\hat{\mathbf{f}}(\mathbf{x})}}}},$$

$$\dot{V}(\mathbf{x},\mathbf{u},\mathbf{d})=$$

$$+(\underset{\mathbf{a}(\mathbf{x})}{\underbrace{\mathbf{A}(\mathbf{x}{)}^{\top }\nabla V(\mathbf{x})}}{)}^{\top }\mathbf{u}+\underset{b(\mathbf{x})}{\underbrace{\mathbf{b}(\mathbf{x}{)}^{\top }\nabla V(\mathbf{x})}},$$

$$\dot{V}(\mathbf{x},\mathbf{u},\mathbf{d})=\stackrel{\hat{\dot{V}}(\mathbf{x},\mathbf{u})}{\overbrace{(\hat{\mathbf{f}}(\mathbf{x})+\hat{\mathbf{g}}(\mathbf{x})\mathbf{u}{)}^{\top }\nabla V(\mathbf{x})+\hat{\mathbf{a}}(\mathbf{x}{)}^{\top }\mathbf{u}+\hat{b}(\mathbf{x})}}$$

Full text

I INTRODUCTION

Properly characterizing uncertainty is a key aspect of robust control [35]. With the increasing use of learning for dynamics modelling and control synthesis [6, 11, 9, 12, 4, 31, 25], it is correspondingly important to develop new tools to reason about the interplay between learning and robust control.

In this paper, we focus on the interplay between learning and robustness for control synthesis using Control Lyapunov Functions (CLFs) [5, 19]. The use of CLFs has seen multiple applications in recent years [20, 15, 24], and one of their primary benefits is to enable control objectives to be represented in a low-dimensional form that can be integrated with optimization methods to yield optimal controllers [3]. This low-dimensional form is also appealing from a learning perspective, as learning is typically more tractable in lower-dimensional spaces [32, 34, 31].

The practical design of CLFs remains challenging. In many cases, extensive tuning upon deployment is necessary [20], and even with this tuning the system is often not able to track a desired state or trajectory perfectly. Other approaches, such as those based on adaptive control [18], can adaptively learn a CLF but are restricted to learning over specific classes of model uncertainty.

We thus build upon ideas in robust control in order to guarantee performance in the presence of model mis-specification. The idea of robust CLFs is not new (cf. [14, 13]), but existing analyses focus on the full-dimensional state dynamics, which can be burdensome for learning.

In this paper, we make two main contributions. First, we propose a novel characterization called Projection to State Stability (PSS), which is a variant of the well-studied Input to State Stability (ISS) property [26, 29, 28, 33, 27], but defined on projected dynamics rather than the original state dynamics. Like ISS, PSS provides a tool to characterize tracking error in terms of the magnitude of the disturbance or uncertainty. Unlike ISS, PSS can characterize dynamic uncertainty directly in the derivative of a CLF, thus allowing a low dimensional representation of the uncertainty. In our second contribution, we demonstrate the practicality of PSS by incorporating it into an episodic learning algorithm.

Our paper is organized as follows. Section II reviews CLFs and ISS. Section III defines Projection to State Stability (PSS), and how PSS enables constructing bounds on the state of a system that depend on a projected disturbance. Section IV defines a broad class of model uncertainty for affine control systems, evaluates how this uncertainty impacts the Lyapunov derivative, and demonstrates how to restrict this uncertainty with data to determine if a system is PSS. Section V discusses how episodic learning can be used to improve PSS guarantees in practice, and presents simulation results with an uncertain inverted pendulum model.

II PRELIMINARIES

This section provides a review of Control Lyapunov Functions (CLFs) and Input to State Stability (ISS). These tools will be used in Section III to define Projection to State Stability. This section concludes with a brief discussion of how these definitions must be modified to hold over a restriction of the domain.

Consider a state space $\mathcal{X}\subseteq\mathbb{R}^{n}$ and a control input space $\mathcal{U}\subseteq\mathbb{R}^{m}$. Assume that $\mathcal{X}$ is path-connected and that $\mathbf{0}\in\mathcal{X}$. Consider a system governed by:

[TABLE]

for state $\mathbf{x}\in\mathcal{X}$ and its derivative $\dot{\mathbf{x}}$, control input $\mathbf{u}\in\mathcal{U}$, and dynamics $\mathbf{f}:\mathcal{X}\times\mathcal{U}\to\mathbb{R}^{n}$. In this paper we assume $\mathbf{f}$ is locally Lipschitz continuous. The following definitions, taken from [17], are useful in analyzing stability of (1).

Definition 1** (Class $\mathcal{K}$ Function).**

A continuous function $\alpha:[0,a)\to\mathbb{R}_{+}$, with $a>0$, is class $\mathcal{K}$, denoted $\alpha\in\mathcal{K}$, if it is monotonically (strictly) increasing and satisfies $\alpha(0)=0$. If the domain of $\alpha$ is all of $\mathbb{R}_{+}$ and $\lim_{r\to\infty}\alpha(r)=\infty$, then $\alpha$ is termed radially unbounded and class $\mathcal{K}_{\infty}$.

Definition 2** (Class $\mathcal{KL}$ Function).**

A continuous function $\beta:[0,a)\times\mathbb{R}_{+}\to\mathbb{R}_{+}$, with $a>0$, is class $\mathcal{KL}$, denoted $\beta\in\mathcal{KL}$, if the function $r\mapsto\beta(r,s)\in\mathcal{K}$ for all $s\in\mathbb{R}_{+}$, and the function $s\mapsto\beta(r,s)$ is monotonically non-increasing with $\beta(r,s)\rightarrow 0$ as $s\rightarrow\infty$ for all $r\in[0,a)$.

We note that the strictly increasing nature of Class $\mathcal{K}$ $(\mathcal{K}_{\infty})$ functions permits an inverse Class $\mathcal{K}$ $(\mathcal{K}_{\infty})$ function $\alpha^{-1}:[0,\alpha(a))\to\mathbb{R}_{+}$. We also note that the composition of Class $\mathcal{K}$ ($\mathcal{K}_{\infty}$) functions is itself a Class $\mathcal{K}$ ($\mathcal{K}_{\infty}$) function. Given these definitions, we define Control Lyapunov Functions (CLFs) as in [5], [19].

Definition 3** (Control Lyapunov Function).**

A continuously differentiable function $V:\mathcal{X}\to\mathbb{R}_{+}$ is a CLF for $(\ref{eq:controlsys})$ on $\mathcal{X}$ if there exist $\underline{\alpha},\overline{\alpha},\alpha\in\mathcal{K}_{\infty}$ such that:

[TABLE]

for all $\mathbf{x}\in\mathcal{X}$.

If there exists a CLF for a system, then a state-feedback controller $\mathbf{k}:\mathcal{X}\to\mathcal{U}$ can be selected such that $\mathbf{0}$ is a globally asymptotically stable equilibrium point. In particular, for all $\mathbf{x}\in\mathcal{X}$, $\mathbf{k}(\mathbf{x})$ should be chosen such that $\dot{V}(\mathbf{x},\mathbf{k}(\mathbf{x}))\leq-\alpha(\left\|\mathbf{x}\right\|)$. We note that $\underline{\alpha},\overline{\alpha},\alpha$ only need to be Class $\mathcal{K}$ for this definition, but we extend them to $\mathcal{K}_{\infty}$ to simplify later analysis.

To accommodate disturbances or uncertainties, we consider a disturbance space $\mathcal{D}\subseteq\mathbb{R}^{d}$, and a modified system:

[TABLE]

for disturbance $\mathbf{d}\in\mathcal{D}$ and dynamics $\mathbf{f}:\mathcal{X}\times\mathcal{U}\times\mathcal{D}\to\mathbb{R}^{n}$. We again assume $\mathbf{f}$ is locally Lipschitz continuous. The disturbance may be time-varying, state-dependent, and/or input-dependent. We assume that the disturbance is bounded for almost all times $t\geq 0$ (essentially bounded in time). This leads to the definition of ISS and ISS-CLFs as formulated in [26], [29].

Definition 4** (Input to State Stability).**

Given a state-feedback controller $\mathbf{k}:\mathcal{X}\to\mathcal{U}$, a system is Input to State Stable (ISS) if there exist $\beta\in\mathcal{KL}_{\infty}$ and $\gamma\in\mathcal{K}_{\infty}$ such that the solution to (3) satisfies:

[TABLE]

for all $t\geq 0$.

Definition 5** (Input to State Stable Control Lyapunov Function).**

A continuously differentiable function $V:\mathcal{X}\to\mathbb{R}_{+}$ is an Input to State Stable Control Lyapunov Function (ISS-CLF) for (3) on $\mathcal{X}$ if there exist $\underline{\alpha},\overline{\alpha},\alpha,\rho\in\mathcal{K}_{\infty}$ such that:

[TABLE]

for all $\mathbf{x}\in\mathcal{X}$ and $\mathbf{d}\in\mathcal{D}$.

As with CLFs, if there exists an ISS-CLF for a system, then a state-feedback controller $\mathbf{k}:\mathcal{X}\to\mathcal{U}$ can be chosen such that the system is ISS. If the disturbance is input-dependent, it is additionally required that $\mathbf{k}$ induces essentially bounded disturbances in time.

The condition on the Lyapunov function derivative in (3) or (5) may not be satisfied on the entire state space $\mathcal{X}$. In particular it may only be satisfied on a subset $\mathcal{C}\subseteq\mathcal{X}$. The system may leave $\mathcal{C}$ during its evolution, implying the desired derivative condition may no longer be satisfiable. We therefore consider the following definition and lemma.

Definition 6** (Forward Invariance).**

Consider the system governed by (1). A subset $\mathcal{F}\subseteq\mathcal{X}$ is forward invariant if there exists a state-feedback controller $\mathbf{k}:\mathcal{X}\to\mathcal{U}$ such that $\mathbf{x}(0)\in\mathcal{F}$ implies $\mathbf{x}(t)\in\mathcal{F}$ for all $t\geq 0$.

The definition of forward invariance applies to systems governed by (3), with disturbances appropriately restricted to subsets of $\mathcal{D}$ if the disturbances are modeled as state-dependent and/or input-dependent. If $\mathbf{0}\in\mathcal{C}$, we may restrict Definitions 3 and 5 to a forward invariant subset $\mathcal{F}\subseteq\mathcal{C}$ with $\mathbf{0}\in\mathcal{F}$, provided such a subset exists.

Lemma 1**.**

A sublevel set $\Omega\subseteq\mathcal{X}$ of an ISS-CLF $V$ is a forward invariant set, provided $\left\|\mathbf{x}\right\|\geq\rho(\left\|\mathbf{d}\right\|)$ for all $\mathbf{x}\in\partial\Omega$ and appropriately restricted $\mathbf{d}\in\mathcal{D}$.

Proof.

The condition on the Lyapunov derivative in (5) implies the existence of a state-feedback controller $\mathbf{k}:\mathcal{X}\to\mathcal{U}$ satisfying $\dot{V}(\mathbf{x},\mathbf{k}(\mathbf{x}),\mathbf{d})\leq-\alpha(\left\|\mathbf{x}\right\|)$ for all $\mathbf{x}\in\partial\Omega$ and appropriately restricted $\mathbf{d}\in\mathcal{D}$. Let $c=V(\mathbf{x})$ for any $\mathbf{x}\in\partial\Omega$. If $V(\mathbf{x}(0))\in[0,c]$, then $V(\mathbf{x}(t))\in[0,c]$ for all $t>0$ by Nagumo’s Theorem [23], [1]. Thus, if $\mathbf{x}(0)\in\Omega$, then $\mathbf{x}(t)\in\Omega$ for all $t\geq 0$. ∎

III Projection to State Stability

Input to State Stability (ISS) requires a bound on the state in terms of the norm of the disturbance as it appears in the state dynamics (see Definition 4 in Section II). This requirement does not easily permit analysis of Input to State behavior when the disturbance is more easily described by its impact in a Lyapunov function derivative. This limitation motivates Projection to State Stability (PSS), which instead relies a bound on the state in terms of a projection of the disturbance.

Definition 7** (Dynamic Projection).**

A continuously differentiable function $\boldsymbol{\Pi}:\mathcal{X}\to\mathbb{R}^{k}$ is a dynamic projection if there exist $\underline{\sigma},\overline{\sigma}\in\mathcal{K}_{\infty}$ satisfying:

[TABLE]

for all $\mathbf{x}\in\mathcal{X}$.

Let $\mathcal{Y}=\mathrm{range}(\boldsymbol{\Pi})$, and let $\mathbf{y}=\boldsymbol{\Pi}(\mathbf{x})$ for all $\mathbf{x}\in\mathcal{X}$. Consider the system governed by (3). The associated projected system is governed by the dynamics:

[TABLE]

where $\mathbf{D}_{\boldsymbol{\Pi}}:\mathcal{X}\to\mathbb{R}^{k\times n}$ denotes the Jacobian of $\boldsymbol{\Pi}$, and $\boldsymbol{\delta}$ is implicitly a function of $\mathbf{x}$, $\mathbf{u}$, and $\mathbf{d}$. For the following definitions, we assume $\boldsymbol{\delta}$ is essentially bounded in time.

We are now ready to state our main definition. The key difference between PSS and ISS (Definition 4) is the use of $\boldsymbol{\delta}$ (7) rather than the native disturbance $\mathbf{d}$.

Definition 8** (Projection to State Stability).**

Given a state-feedback controller $\mathbf{k}:\mathcal{X}\to\mathcal{U}$, a system is Projection to State Stable (PSS) with respect to the projection $\boldsymbol{\Pi}$ if there exist $\beta\in\mathcal{KL}_{\infty}$ and $\gamma\in\mathcal{K}_{\infty}$ such that the solution to (3) satisfies:

[TABLE]

for all $t\geq 0$, with $\boldsymbol{\delta}$ as defined in (7).

Remark 1*.*

If $\boldsymbol{\Pi}$ is an inclusion map with $k=n$, and the system can be specified as:

[TABLE]

then PSS is equivalent to ISS.

Similarly, we can also construct a Lyapunov function that certifies a system is PSS with respect to a projection.

Definition 9** (Projection to State Stable Control Lyapunov Function).**

A continuously differentiable function $W:\mathcal{Y}\to\mathbb{R}_{+}$ is a Projection to State Stable Control Lyapunov Function (PSS-CLF) for (7) on $\mathcal{X}$ if there exist $\underline{\alpha},\overline{\alpha},\alpha,\rho\in\mathcal{K}_{\infty}$ satisfying:

[TABLE]

for all $\mathbf{x}\in\mathcal{X}$.

As with ISS-CLFs, this definition can be restricted to a forward invariant set containing $\mathbf{0}$. We now show that a PSS-CLF certifies a system is PSS.

Theorem 1**.**

If the system governed by (7) has a PSS-CLF, then the system governed by (3) is PSS with respect to the projection $\boldsymbol{\Pi}$.

Proof.

The bounds in (9) can be weakened to:

[TABLE]

That is, if (III) holds, (9) holds. Therefore, a choice of state-feedback controller exists such that the system governed by (7) is Input to State Stable (ISS) with $\boldsymbol{\delta}$ viewed a disturbance. This implies that there exist $\beta\in\mathcal{KL}_{\infty}$ and $\gamma\in\mathcal{K}_{\infty}$ such that:

[TABLE]

for all $t\geq 0$. Since $\boldsymbol{\Pi}$ satisfies (6) we have:

[TABLE]

Finally, define $\beta^{\prime}\in\mathcal{KL}_{\infty}$ and $\gamma^{\prime}\in\mathcal{K}_{\infty}$ as:

[TABLE]

From the weak form of the triangle inequality presented in [26], [16], it follows that:

[TABLE]

∎

We next show that a CLF $V$ for the undisturbed dynamics of a system can be viewed as a projection, thus yielding a PSS-CLF that certifies PSS with respect to $V$.

Corollary 1**.**

Suppose $V:\mathcal{X}\to\mathbb{R}_{+}$ is a CLF on $\mathcal{X}$ for the system $\dot{\mathbf{x}}=\mathbf{f}(\mathbf{x},\mathbf{u},\mathbf{0}).$ Then the disturbed system governed by (3) is PSS with respect to the projection $V$.

Proof.

With the projection $V$ we have that:

[TABLE]

where $\nabla V:\mathcal{X}\to\mathbb{R}^{n}$ is the gradient of the Lyapunov function. The projected system is governed by:

[TABLE]

Since $V$ is a CLF, there exists a state-feedback controller $\mathbf{k}:\mathcal{X}\to\mathcal{U}$ satisfying:

[TABLE]

for all $\mathbf{x}\in\mathcal{X}$. Let $\alpha_{p},\alpha_{q}\in\mathcal{K}_{\infty}$ satisfy $\alpha_{p}+\alpha_{q}=\mathbf{\alpha}$. Then:

[TABLE]

Therefore:

[TABLE]

Since $V$ is a CLF we may weaken the bounds as in the proof of Theorem 1 to:

[TABLE]

noting that $\overline{\alpha}\circ\alpha_{q}^{-1}$ and $\alpha_{p}\circ\overline{\alpha}^{-1}$ are class $\mathcal{K}_{\infty}$. It follows from Definition 9 that the identity map on $\mathbb{R}_{+}$ is a PSS-CLF for (18). Therefore, the system (3) is PSS with respect to the projection $V$ by Theorem 1. ∎

IV UNCERTAINTY MODELING & ANALYSIS

In this section we consider a structured form of uncertainty present in affine control systems. We analyze the impact of this uncertainty on a Lyapunov function derivative, and on the PSS behavior of the system.

IV-A Uncertain Affine Systems

We consider affine control systems of the form:

[TABLE]

with drift dynamics $\mathbf{f}:\mathcal{X}\to\mathbb{R}^{n}$ and actuation matrix $\mathbf{g}:\mathcal{X}\to\mathbb{R}^{n\times m}$. If $\mathbf{f}$ and $\mathbf{g}$ are unknown, we may consider an estimated model of the system:

[TABLE]

where $\hat{\mathbf{f}}:\mathcal{X}\to\mathbb{R}^{n}$ and $\hat{\mathbf{g}}:\mathcal{X}\to\mathbb{R}^{n\times m}$ are estimates of $\mathbf{f}$ and $\mathbf{g}$, respectively. In this case, (23) can be expressed as:

[TABLE]

obtaining a representation of the dynamics as in (9). Note that the disturbance $\mathbf{d}=\mathbf{A}(\mathbf{x})\mathbf{u}+\mathbf{b}(\mathbf{x})$ is explicitly characterized as time-invariant, state-dependent, and input-dependent, with potentially unknown $\mathbf{A}(\mathbf{x})$ and $\mathbf{b}(\mathbf{x})$ for all $\mathbf{x}\in\mathcal{X}$.

As discussed in [2], [31], CLFs may be constructively formed for affine systems under proper assumptions regarding relative degree and unbounded control. Furthermore, if the true system satisfies the relative degree properties of the estimated model, then the CLF found for the estimated system can be used for the true system.

Assume $\mathbf{f}$, $\mathbf{g}$, $\hat{\mathbf{f}}$, and $\hat{\mathbf{g}}$ are Lipschitz continuous (implying $\mathbf{A}$ and $\mathbf{b}$ are Lipschitz continuous), and let $V$ be a CLF candidate for (24). The time derivative of $V$ is given by:

[TABLE]

for all $\mathbf{x}\in\mathcal{X}$ and $\mathbf{u}\in\mathcal{U}$. As proposed in [31], we may wish to reduce the estimation error $|\dot{V}-\hat{\dot{V}}|$ by improving $\hat{\dot{V}}$ with estimates of $\mathbf{a}$ and $b$. Given continuous estimators $\hat{\mathbf{a}}:\mathcal{X}\to\mathbb{R}^{m}$ and $\hat{b}:\mathcal{X}\to\mathbb{R}$, (IV-A) may be reformulated as:

[TABLE]

for all $\mathbf{x}\in\mathcal{X}$ and $\mathbf{u}\in\mathcal{U}$.

Both formulations decompose $\dot{V}$ into an estimated component, $\hat{\dot{V}}$, and a residual component. In (IV-A) the residual terms $\mathbf{a}$ and $b$ capture the effect of the unmodeled dynamics on the Lyapunov function derivative. In (IV-A) the residual terms reflect the error in estimating this effect. Additionally, viewing $V$ as a projection results in $\delta=\mathbf{a}(\mathbf{x})^{\top}\mathbf{u}+b(\mathbf{x})$.

IV-B Projection to State Stability via Uncertainty Functions

If knowledge on what values $\mathbf{a}$ and $b$ can assume is available, the impact on the Lyapunov derivative can be constrained in a manner permitting PSS analysis of a system. Therefore, we define a function characterizing the possible uncertainties at a given state.

Definition 10** (Uncertainty Function).**

Let $\mathcal{P}(\mathbb{R}^{m}\times\mathbb{R})$ denote the set of all subsets of $\mathbb{R}^{m}\times\mathbb{R}$. An uncertainty function for (IV-A) or (IV-A) is a function $\Delta:\mathcal{X}\to\mathcal{P}(\mathbb{R}^{m}\times\mathbb{R})$ with $\Delta(\mathbf{x})$ bounded and satisfying $(\mathbf{a}(\mathbf{x}),b(\mathbf{x}))\in\Delta(\mathbf{x})$ for all $\mathbf{x}\in\mathcal{X}$.

For a given $\mathbf{x}\in\mathcal{X}$, we refer to $\Delta(\mathbf{x})$ as an uncertainty set. Suppose there exists a valid uncertainty function $\Delta$ for (IV-A) or (IV-A). Then $V$ satisfies:

[TABLE]

for all $\mathbf{x}\in\mathcal{X}$ and $\mathbf{u}\in\mathcal{U}$. One major challenge is to define a $\Delta$ that is non-vacuous and thus practically relevant. From this point forward we limit our attention to a subset of the state space and make a critical assumption regarding the estimate $\hat{\dot{V}}$ for a CLF $V$.

Assumption 1**.**

Let $V$ be a CLF for the system governed by (24) on a subset $\mathcal{C}\subseteq\mathcal{X}$ with $\mathbf{0}\in\mathcal{C}$. We assume that:

[TABLE]

for all $\mathbf{x}\in\mathcal{C}$. If $\hat{\dot{V}}$ is specified as in (IV-A), then this assumption is satisfied by definition. If $\hat{\dot{V}}$ is specified as in (IV-A), then this assumption states that the addition of the estimators $\hat{\mathbf{a}}$ and $\hat{b}$ does not make it impossible to choose a control input such that (29) is satisfied.

If the estimated and true system satisfy the same relative degree property, then this assumption amounts to the addition of estimates $\hat{\mathbf{a}}$ and $\hat{b}$ not violating the relative degree property.

Assumption 2**.**

Let $\mathbf{A}$ and $\mathbf{b}$ be defined as in (25), and let $\mathcal{C}$ be defined as in Assumption 1. We assume $\mathbf{A}$ and $\mathbf{b}$ are bounded on $\mathcal{C}$.

If $\mathcal{C}$ is compact, this assumption is automatically satisfied since $\mathbf{A}$ and $\mathbf{b}$ are assumed continuous. Under Assumption 1, the set of admissible control inputs $\mathcal{U}(\mathbf{x})$:

[TABLE]

is non-empty, for all $\mathbf{x}\in\mathcal{C}$. Then the CLF $V$ satisfies:

[TABLE]

for all $\mathbf{x}\in\mathcal{C}$. We now develop sufficient conditions on the uncertainty function that certifies (25) as PSS with respect to the CLF $V$ (with $V$ interpreted as a projection).

Theorem 2** (Sufficient Conditions for PSS in Affine Control Systems).**

Consider the system in (25), and a CLF $V$ for (24) with estimated time-derivative $\hat{\dot{V}}$ as defined in (IV-A) or (IV-A), satisfying Assumption 1. Let $\Delta$ be an uncertainty function and let $\mathbf{k}:\mathcal{X}\to\mathcal{U}$ be a state-feedback controller satisfying $\mathbf{k}(\mathbf{x})\in\mathcal{U}(\mathbf{x})$ for all $\mathbf{x}\in\mathcal{C}$, with $\mathcal{U}(\mathbf{x})$ defined as in (30). Suppose there exists $\alpha_{p},\alpha_{q}\in\mathcal{K}_{\infty}$ with $\alpha_{p}+\alpha_{q}=\alpha$ and a sublevel set $\Omega\subseteq\mathcal{C}$ of $V$ satisfying:

[TABLE]

*for all $\mathbf{x}\in\partial\Omega$. Then the system governed by (25) is PSS with respect to the projection $V$ on $\Omega$. *

Proof.

First, note that:

[TABLE]

for all $\mathbf{x}\in\mathcal{C}$. Since (32) holds for all $\mathbf{x}\in\partial\Omega$ and $\alpha_{q}$ is monotonically increasing, we have:

[TABLE]

for all $\mathbf{x}\in\partial\Omega$. It follows that:

[TABLE]

for all $\mathbf{x}\in\partial\Omega$. This means $\Omega$ is forward invariant, with a proof similar to that of Lemma 1. Since $V$ is a CLF for (24), Corollary 1 can be restricted to $\Omega$; that is, the system is PSS with respect to the projection $V$ on $\Omega$. ∎

We may want to study a particular set of interest $\mathcal{E}$ over which the impact of the uncertainty can be bounded. For $r>0$, let $B_{r}$ be the open ball around $\mathbf{0}$ of radius $r$, typically used to define a ball contained in $\mathcal{E}$ in the subsequent analysis.

Corollary 2**.**

Suppose there is a set $\mathcal{E}$ and $\mu\geq 0$ satisfying:

[TABLE]

for all $\mathbf{x}\in\mathcal{E}$. If there exists a sublevel set $\Omega$ of $V$ such that:

[TABLE]

then the system is PSS with respect to the (CLF) projection $V$ on $\Omega$, and the smallest sublevel set of $V$ containing $B_{\alpha_{q}^{-1}(\mu)}$ is asymptotically stable.

Proof.

First, note that:

[TABLE]

for all $\mathbf{x}\in\partial\Omega$, and the system is PSS on $\Omega$ by Theorem 2. The smallest sublevel set of $V$ containing $B_{\alpha_{q}^{-1}(\mu)}$ is asymptotically stable since:

[TABLE]

∎

Improving the uncertainty set (e.g., reducing uncertainty using learning) directly leads to larger sets for a given bound, or tighter bounds on a given set. We state this formally in the next result.

Corollary 3** (Uncertainty Function Improvement).**

Consider uncertainty functions $\Delta$ and $\Delta^{\prime}$, as well as $\mathcal{E}$ and $\mu$ as defined in Corollary 2.

• •

Fix $\mu>0$ and let $\mathcal{E}_{\mu}$ be defined as:

[TABLE]

• •

Fix $\mathcal{E}\subseteq\mathcal{X}$ and let $\mu_{\mathcal{E}}$ be defined as:

[TABLE]

Suppose $\Delta^{\prime}(\mathbf{x})\subseteq\Delta(\mathbf{x})$ for all $\mathbf{x}\in\mathcal{X}$. Then the associated set $\mathcal{E}_{\mu}^{\prime}$ and scalar $\mu_{\mathcal{E}}^{\prime}$ satisfy $\mathcal{E}_{\mu}\subseteq\mathcal{E}_{\mu}^{\prime}$ and $\mu_{\mathcal{E}}^{\prime}\leq\mu_{\mathcal{E}}$.

Proof.

[TABLE]

∎

IV-C Uncertainty Function Construction

We now provide a constructive method for creating an uncertainty function from a dataset of of state and control values generated by a system. Assume $\mathbf{A}$ and $\mathbf{b}$ are Lipschitz continuous with constants $L_{\mathbf{A}}$ and $L_{\mathbf{b}}$, respectively. Additionally, assume that $\mathbf{A}$ and $\mathbf{b}$ are bounded on $\mathcal{C}$ by constants $\left\|\mathbf{A}\right\|_{\infty}$ and $\left\|\mathbf{b}\right\|_{\infty}$, respectively. Consider a dataset $D\subseteq(\mathcal{X}\times\mathcal{U})\times\mathbb{R}$ consisting of data-measurement pairs $((\mathbf{x},\mathbf{u}),\dot{V}(\mathbf{x},\mathbf{u},\delta))$. Such measurements of $\dot{V}$ can be obtained through numerical differentiation of computed values of $V$. For notational convenience, let $D_{0}=\{(\mathbf{x},\mathbf{u}):((\mathbf{x},\mathbf{u}),\dot{V}(\mathbf{x},\mathbf{u},\delta))\in D\}$.

Proposition 1**.**

Given a dataset $D$, an uncertainty function $\Delta$ can be constructed as:

[TABLE]

for all $\mathbf{x}\in\mathcal{X}$, where $\epsilon:\mathcal{X}\times\mathcal{X}\times\mathcal{U}\to\mathbb{R}_{+}$ is continuous.

Remark 2*.*

For all $\mathbf{x}\in\mathcal{X}$, $\Delta(\mathbf{x})$ is a closed, symmetric polyhedron and is bounded given sufficiently diverse control inputs in the dataset. In this case, $\Delta(\mathbf{x})$ is a compact, convex set. The supremum present in Theorem 2 and Corollary 2 becomes a linear program (LP) and can be efficiently solved.

Proof of Proposition 1.

Define observed error as:

[TABLE]

for all $(\mathbf{x},\mathbf{u})\in D_{0}$. Consider a test point $(\mathbf{x},\mathbf{u})\in\mathcal{X}\times\mathcal{U}$ and a data point $(\mathbf{x}^{\prime},\mathbf{u}^{\prime})\in D_{0}$. Note that $\ell(\mathbf{x}^{\prime},\mathbf{u}^{\prime})$ satisfies:

[TABLE]

where the inequality follows from the reverse triangle inequality, triangle inequality, and Cauchy-Schwarz inequality.

For simplicity we proceed with the construction assuming the estimated Lyapunov function derivative is specified as in (IV-A). The resulting bound will be modified to include estimators as in (IV-A). Note that:

[TABLE]

where the inequality follows from the triangle inequality, submultiplicativity of matrix norms, and Lipschitz and bounded assumptions for $\bf{A}$. Since it is also true that:

[TABLE]

then the following bound holds:

[TABLE]

Let $\epsilon_{L}(\mathbf{x},\mathbf{x}^{\prime})=\left\|\mathbf{x}-\mathbf{x}^{\prime}\right\|\min{\{\left\|\nabla V(\mathbf{x})\right\|_{2},\left\|\nabla V(\mathbf{x}^{\prime})\right\|_{2}\}}$, and let $\epsilon_{\infty}(\mathbf{x},\mathbf{x}^{\prime})=\left\|\nabla V(\mathbf{x})-\nabla V(\mathbf{x}^{\prime})\right\|_{2}$. Observe that $\epsilon_{L}$ and $\epsilon_{\infty}$ are continuous functions. Next, note that:

[TABLE]

Similarly,

[TABLE]

Therefore,

[TABLE]

While $\epsilon_{L}$ and $\epsilon_{\infty}$ decrease as the test point approaches data points, without estimators as in (IV-A), the observed loss term $\ell$ can remain large. By including such estimators, the observed loss term may be reduced, but the bound must be modified with the following additional continuous function:

[TABLE]

which accounts for potential error in the estimation at the test point. $\epsilon$ is then specified as the total upper bound. ∎

Corollary 4**.**

The uncertainty set $\Delta$ as specified in (1) is continuous with respect to the Hausdorff metric.

Proof.

Note that the inequality constraints in (1) can be rewritten as:

[TABLE]

where $\boldsymbol{\Xi}\in\mathbb{R}^{2|D|\times(m+1)}$, $\boldsymbol{\xi}:\mathcal{X}\to\mathbb{R}^{2|D|}$ and $\preceq$ denotes elementwise inequality. The function $\boldsymbol{\xi}$ is continuous since $\epsilon$ is continuous; therefore, the results established in [8] show that the point-to-set map $\Delta$ is continuous. ∎

V INTEGRATION WITH LEARNING

We now explore the practical interplay between learning and systematic improvement of PSS properties, in particular by decreasing the upper bound in (IV-C). By decreasing this bound, the uncertainty set in (1) can be made smaller, which in turn can increase the state space region over which PSS properties can be certified and/or achieve reduced degradation (see Corollary 3). As discussed previously, using PSS rather than ISS enables lower-dimensional learning objectives and upper bounds which can be efficiently evaluated during and after learning.

Learning also offers direct ways to decrease the upper bound in (IV-C). As discussed in Section IV, estimators can be used to reduce the observed loss in (44), which appears directly in the upper bound. We can use supervised learning to train such estimators. One complication is that, using baseline controllers, it may not be possible to collect data in regions we wish to certify PSS properties. As the distances between a point of interest and previously collected data grow, $\epsilon_{L}$ and $\epsilon_{\infty}$ can grow larger, weakening uncertainty bounds at the point of interest. By refining the baseline controller using learned models, the system may be controlled towards these regions of interest.

V-A Episodic Learning Framework

We demonstrate the practicality of PSS by incorporating it into an episodic learning framework based on learning CLF time derivatives [31]. Controller improvement is achieved by alternating between executing a controller to gather data and refining estimates of residual uncertainty. As data collection and learning progresses, the size of uncertainty sets decreases, enabling stronger PSS certifications for the system.

We briefly describe the DaCLyF (Dataset Aggregation for Control Lyapunov Functions) learning approach from [31]. Let $\mathcal{H}_{\mathbf{a}}$ and $\mathcal{H}_{b}$ be nonlinear estimator classes, and let $\mathcal{H}$ be the class of estimators of the form:

[TABLE]

for all $\mathbf{x}\in\mathcal{X}$, given estimators $\hat{\mathbf{a}}\in\mathcal{H}_{\mathbf{a}}$ and $\hat{b}\in\mathcal{H}_{b}$. Equipped with a loss function $\mathcal{L}:\mathbb{R}\times\mathbb{R}\to\mathbb{R}_{+}$ and a dataset $D=\{((\mathbf{x}_{i},\mathbf{u}_{i}),\dot{V}_{i})\}$ obtained from experiments with a baseline controller, we approximately solve the Empirical Risk Minimization (ERM) problem over the class $\mathcal{H}$ to update the estimate of the Lyapunov function derivative. Then, the controller is updated with an augmenting controller of the form:

[TABLE]

for all $\mathbf{x}\in\mathcal{C}$, with $\mathbf{P}\in\mathbb{S}^{2m}_{+}$, $\mathbf{q}\in\mathbb{R}^{2m}$, $r>0$, and $\hat{\dot{V}}$ denoting the updated estimator. Here $\mathbb{S}^{2m}_{+}$ denotes the set of positive semidefinite matrices of size $2m\times 2m$. The augmenting controller, weighted by a trust factor, is additively incorporated with the baseline controller. As more data is collected, the trust factor is increased. This entire procedure is outlined in Algorithm 1.

The estimator class $\mathcal{H}$ can include a wide variety of nonlinear functions. Should $\mathcal{H}_{\mathbf{a}}$ and $\mathcal{H}_{b}$ be classes of Lipschitz continuous estimators, the upper bound (52) can be weakened further using the associated Lipschitz constants to permit further analysis of the uncertainty function specified in (1). Importantly, this motivates the use of spectrally normalized deep neural networks [21], [7] for this estimation problem.

V-B Simulation Results

In this section we apply Algorithm 1 to an inverted pendulum model with parametric uncertainty. The pendulum is modeled as a massless rod with torque input at a fixed base. The true mass and the length are perturbed by up to 30% of their estimated values. The baseline controller is a linear proportional derivative (PD) controller to track angle and angle rate trajectories. The estimators are chosen from the class of two layer neural networks with 200 hidden units and ReLU nonlinearities, mapping concatenated state and Lyapunov function gradients to $\mathbb{R}^{m}$ and $\mathbb{R}$. The trust factors are chosen in a sigmoid fashion. Naive exploratory control is introduced as in [31], with perturbations chosen uniformly at random, independently in each coordinate, and scaled by 25% of the norm of the current control input.

A comparison of the baseline controller and final augmented controller demonstrating improved tracking performance is shown in Fig. 1. A comparison of PSS bounds for the model-based QP controller and the final augmented controller is shown in Fig. 2, with observed trajectories superimposed. The QP controller is unable to keep the system in regions in which the bound is shown to be small. On the other hand, the augmented controller keeps the system close to the desired trajectory, consistently near training data. The bounds are small along the observed trajectory, in comparison.

VI CONCLUSION

We presented a novel low-dimensional view of stability for uncertain systems and a method of evaluating PSS behavior using experimental data. This method constructs a bound on disturbances to a CLF derivative, and can be integrated with a machine learning framework to improve PSS behavior. Finally, we validate this procedure on a simulated system.

Future work includes incorporating the upper bounds into online learning settings and developing optimal exploration strategies. Quantifying the impact of learning on PSS provides an objective for deciding how to collect data, also known as the exploration problem in learning literature [22, 6, 10, 9, 30]. In particular, reductions of the uncertainty bound may be used to formulate regret in online learning settings or reward in imitation and reinforcement learning settings. Additional future work includes extending the notion of PSS to Control Safety and Barrier Functions, more thoroughly studying the benefits of learning low-dimensional representations of the dynamics versus the full-order dynamics, and utilizing PSS to augment controller synthesis for complex real-world robotic systems.