Learning Lyapunov (Potential) Functions from Counterexamples and Demonstrations

TL;DR

This paper introduces an iterative learning framework that synthesizes control Lyapunov functions for nonlinear systems by leveraging demonstrations, counterexamples, and convex optimization, enabling the replacement of complex MPC controllers with simpler, guaranteed controllers.

Contribution

It presents a novel iterative learning approach combining demonstrations, counterexamples, and verification to synthesize control Lyapunov functions for nonlinear systems.

Findings

Successfully synthesizes polynomial control Lyapunov functions.

Replaces MPC controllers with simpler, guaranteed controllers.

Proves convergence of the learning algorithm using convex optimization techniques.

Abstract

We present a technique for learning control Lyapunov (potential) functions, which are used in turn to synthesize controllers for nonlinear dynamical systems. The learning framework uses a demonstrator that implements a black-box, untrusted strategy presumed to solve the problem of interest, a learner that poses finitely many queries to the demonstrator to infer a candidate function and a verifier that checks whether the current candidate is a valid control Lyapunov function. The overall learning framework is iterative, eliminating a set of candidates on each iteration using the counterexamples discovered by the verifier and the demonstrations over these counterexamples. We prove its convergence using ellipsoidal approximation techniques from convex optimization. We also implement this scheme using nonlinear MPC controllers to serve as demonstrators for a set of state and trajectory stabilization problems for nonlinear dynamical systems. Our approach is able to synthesize relatively simple polynomial control Lyapunov functions, and in that process replace the MPC using a guaranteed and computationally less expensive controller.

Figures

Tables

Table 1. TABLE I: Results on the numerical examples. n 𝑛 n : # variables, m 𝑚 m : # control inputs, τ 𝜏 \tau : MPC time step, ℋ ℋ \mathcal{H} : MPC time horizon, D V subscript 𝐷 𝑉 D_{V} : SDP relaxation degree bound for the Verifier , #Itr: # of iterations, Time: total computation time (minutes).

System Name	$n$	$m$	$\tau$	$\mathscr{H}$	$D_V$	# Itr	Time
Tora	4	1	1	30	4	53	30
Bicycle	4	2	0.4	8	3	51	9
Bicycle $\times$ 2	8	4	0.4	8	3	536	303
Inverted Pendulum	4	1	0.04	2	5	85	31
Forward Flight	4	2	0.4	16	5	32	26
Hover Mode	6	2	0.4	16	4	213	163
Unicycle (Seg.1)	3+1*	2	0.1	2	4	41	15
Unicycle (Seg.2)	3+1*	2	0.1	3	4	31	7

Equations

$$\text{accentset}\text{mbox}.x_1=x_2,\text{accentset}\text{mbox}.x_2=-x_1+ϵ\sin (x_3),\text{accentset}\text{mbox}.x_3=x_4,\text{accentset}\text{mbox}.x_4=u_1.$$

$$\sum _{t=[0,\tau ,...,\mathcal{H}]}\left(||\mathbf{x}(t)|{|}_2^2+||\mathbf{u}(t)|{|}_2^2\right)+\mathcal{H}||\mathbf{x}(\mathcal{H})|{|}_2^2.$$

$${\small\left(\begin{array}[]{c}1.22\ x_{2}^{2}+0.31\ x_{2}x_{3}\ +0.44\ x_{3}^{2}-0.28\ x_{4}x_{2}+0.8\ x_{4}x_{3}\\ +1.69\ x_{4}^{2}+0.069\ x_{1}x_{2}-0.68\ x_{1}x_{3}-1.85\ x_{4}x_{1}+1.60\ x_{1}^{2}\end{array}\right)}$$

$$\text{accentset}\text{mbox}.\mathbf{x}=f(\mathbf{x},\mathbf{u})=f_0(\mathbf{x})+\sum _{i=1}^m{f}_i(\mathbf{x})u_i,$$

$$\begin{array}[]{rl}&V(\mathbf{0})=0\\ (\forall\mathbf{x}\in S\setminus\{0\})&V(\mathbf{x})>0\\ (\forall\mathbf{x}\in S\setminus\{0\})\ (\exists\mathbf{u}\in U)&\nabla V\cdot f(\mathbf{x},\mathbf{u})<0\,.\\ \end{array}$$

$$V^{\bowtie \beta }=\{\mathbf{x}|V(\mathbf{x})\bowtie \beta \},\text{mbox}where\bowtie \in \{=,\le ,<,\ge ,>\}.$$

$$V(\mathbf{x}(t_1))=\left(V(\mathbf{x}(0))+{\int }_0^{t_1}\frac{dV}{dt}dt\right)<V(\mathbf{x}(0)).$$

$$V_{\mathbf{c}}(\mathbf{x})=\sum _{j=1}^r{c}_j{g}_j(\mathbf{x}),$$

$$W_j:\{({\mathbf{x}}_1,{\mathbf{u}}_1),\dots ,({\mathbf{x}}_j,{\mathbf{u}}_j)\}\subseteq S\times U.$$

$$C_{j}:\left\{\mathbf{c}\in C\ \left|\ \mathop{\bigwedge}_{(\mathbf{x}_{i},\mathbf{u}_{i})\in W_{j}}\begin{array}[]{c}\ V_{\mathbf{c}}(\mathbf{x}_{i})>0\ \land\\ \nabla V_{c}\cdot f(\mathbf{x}_{i},\mathbf{u}_{i})<0\end{array}\ \right.\right\}\,.$$

$$C_{j+1}:\ C_{j}\cap\left\{\mathbf{c}\ |\ \begin{array}[]{c}V_{\mathbf{c}}(\mathbf{x}_{j+1})>0\ \land\\ \nabla V_{\mathbf{c}}\cdot f(\mathbf{x}_{j+1},\mathbf{u}_{j+1})<0\end{array}\right\}\,.$$

$$V_j({\mathbf{x}}_{j+1})>0\wedge \nabla V_j\cdot f({\mathbf{x}}_{j+1},{\mathbf{u}}_{j+1})<0.$$

$$V_j({\mathbf{x}}_{j+1})\le 0\vee \nabla V_j\cdot f({\mathbf{x}}_{j+1},{\mathbf{u}}_{j+1})\ge 0,$$

$$(\forall \mathbf{x}\in S\setminus \{0\})\nabla V\cdot f(\mathbf{x},\mathcal{D}(\mathbf{x}))<0.$$

$$C_j^1:$$

$$C_j^2:$$

$$(\forall ({\mathbf{x}}_i,{\mathbf{u}}_i=\mathcal{D}({\mathbf{x}}_i))\in W_j)\nabla V_{\mathbf{c}}\cdot f({\mathbf{x}}_i,{\mathbf{u}}_i)<0.$$

$$(\exists ({\mathbf{x}}_i,{\mathbf{u}}_i)\in W_j)V_{\mathbf{c}}({\mathbf{x}}_i)\le 0,$$

$$V(0)=0$$

$$\displaystyle(\forall\mathbf{x}\in S\setminus\{\mathbf{0}\})\$$

$$C_{j+1}:\ C_{j}\cap\left\{\mathbf{c}\ |\ \begin{array}[]{c}\sum_{i=1}^{r}(c_{i}\ g_{i}(\mathbf{x}_{j+1}))>0\ \land\\ \sum_{i=1}^{r}(c_{i}\ \nabla g_{i}\cdot f(\mathbf{x}_{j+1},\mathbf{u}_{j+1}))<0\end{array}\right\}\,.$$

$$\text{mbox}\text{sc}Vol\left(C_{j+1}\right)\le \left(1-\frac{1}{r}\right)\text{mbox}\text{sc}Vol\left(C_j\right).$$

$$\frac{r(\log (\Delta )-\log (\delta ))}{-\log \left(1-\frac{1}{r}\right)}=O(r^2)\text{mbox}iterations.$$

$$\text{mbox}\text{sc}Vol(C_j)\le (1-\frac{1}{r}{)}^j\text{mbox}\text{sc}Vol(C_0)<(1-\frac{1}{r}{)}^j\gamma {\Delta }^r$$

$$⟹$$

$$\log (\text{mbox}\text{sc}Vol(C_j))-\log (\gamma {\Delta }^r)<\frac{r(\log (\Delta )-\log (\delta ))}{-\log (1-\frac{1}{r})}\log (1-\frac{1}{r})$$

$$⟹$$

$$(\exists \mathbf{x}\in S\setminus \{0\})V_j(\mathbf{x})\le 0.$$

$$(\exists \mathbf{x}\in S\setminus \{0\})(\forall \mathbf{u}\in U)(\nabla V_j)\cdot f(\mathbf{x},\mathbf{u})\ge 0.$$

$$\begin{array}[]{ll}(\exists\ \mathbf{x}\in S\setminus\{\mathbf{0}\},\mathbf{\lambda})&\mathbf{\lambda}\geq\mathbf{0},\mathbf{\lambda}^{t}\mathbf{b}\geq-\nabla V_{j}.f_{0}(\mathbf{x})\\ &A_{i}^{t}\mathbf{\lambda}=\nabla V_{j}.f_{i}(\mathbf{x})(i\in\{1\ldots m\}).\end{array}$$

Full text

I Introduction

We propose a novel learning from demonstration scheme for inferring control Lyapunov functions (potential functions) for stabilizing nonlinear dynamical systems to reference states/ trajectories. Control Lyapunov functions (CLFs) have wide applications to motion planning problems in robotics. They extend the classic notion of Lyapunov functions to systems involving control inputs [3]. Finding a CLF also leads us to an associated feedback control law that can be used to solve the stabilization problem. Additionally, they can be extended for feedback motion planning using extensions to time-varying or sequential CLFs [5, 52]. Likewise, they have been investigated in the robotics community in many forms including artificial potential functions to solve path planning problems involving obstacles [30].

However, synthesizing CLFs for nonlinear systems remains a challenge [41, 42]. Standard approaches to finding CLFs include the use of dynamic programming, wherein the value function satisfies the conditions of a CLF [4], or using non-convex bilinear matrix inequalities (BMI) [13]. BMIs can be solved using alternating minimization methods [8, 50, 31]. However, these approaches often get stuck in local minima and exhibit poor convergence guarantees [12].

In this article, we investigate the problem of learning a CLF using a black-box demonstrator that can be queried with a given system state, and responds by demonstrating control inputs to stabilize the system starting from that state. However, our framework uses just the control input at the query state. Such a demonstrator can be realized using an expensive nonlinear model predictive controller (MPC) that uses a local optimization scheme, or even a human operator under certain assumptions 111We do not handle noisy or erroneous demonstrations in this paper. The framework has a Learner which selects a candidate CLF and a Verifier that tests whether this CLF is valid. If the CLF is invalid, the Verifier returns a state at which the current candidate fails. The Learner queries the demonstrator to obtain a control input corresponding to this state. It subsequently eliminates the current candidate along with a set of related functions from further consideration. The framework continues to exhaust the space of candidate CLFs until no CLFs remain or a valid CLF is found in this process.

We prove the process can converge in finitely many steps provided the Learner chooses the candidate function appropriately at each step. We also provide efficient SDP-based approximations to the verification problem that can be used to drive the framework. Finally, we test this approach on a variety of examples, by solving stabilization problems for nonlinear dynamical systems. We show that our approach can successfully find simple CLFs using finite horizon nonlinear MPC schemes with appropriately chosen cost functions to serve as demonstrators. In these instances, the CLFs yield control laws that are computationally inexpensive, and guaranteed against the original dynamical model.

I-A Illustrative Example: Tora System

Figure 1(a) shows a mechanical system consisting of a cart attached to a wall using a spring. The position of the cart $x$ is controlled by an arm with a weight that can be moved back and forth by applying a force $u$, as shown. The goal is to stabilize the cart to $x=0$, with its velocity, angle, and angular velocity $\accentset{\mbox{\large.}}{x}=\theta=\accentset{\mbox{\large.}}{\theta}=0$. We refer the reader to the Jankovic et al. [17] for a derivation of the dynamics shown below in terms of state variables $(x_{1},\ldots,x_{4})$ and control input $u_{1}$, after a suitable change of basis transformation:

[TABLE]

We approximate $\sin(x_{3})$ using a degree $3$ approximation which is quite accurate over the range $x_{3}\in[-2,2]$. The equilibrium $x=\accentset{\mbox{\large.}}{x}=\theta=\accentset{\mbox{\large.}}{\theta}=0$ now corresponds to $x_{1}=x_{2}=x_{3}=x_{4}=0$. The state space is taken to be $S:[-1,1]\times[-1,1]\times[-2,2]\times[-1,1]$, the control input $u_{1}\in[-1.5,1.5]$.

MPC Scheme:

A first approach to doing so uses a nonlinear model-predictive control (MPC) scheme using the time horizon $\mathcal{H}=30$, time step $\tau=1$ and a simple cost function

[TABLE]

Given the model in (1), such a control is implemented using a first order numerical gradient descent method to minimize the cost function over a finite time horizon. The convergence was informally confirmed by observing hundreds of such simulations from different initial conditions for the system. However, the MPC scheme is expensive, requiring repeated solutions to (constrained) nonlinear optimization problems in real-time. Furthermore, the closed loop lacks formal guarantees despite the high confidence gained from numerous simulations.

Learning a Control Lyapunov Function:

The approach in this paper uses the MPC scheme as a “demonstrator”, and attempts to learn a simpler control law through a control Lyapunov function. The key idea depicted in Fig. 2 is to pose queries to the MPC at finitely many witness states $W=\{\mathbf{x}_{1},\ldots,\mathbf{x}_{j}\}$ and use the corresponding instantaneous control inputs $\mathbf{u}_{1},\ldots,\mathbf{u}_{j}$, respectively. The learner attempts to find a candidate function $V(\mathbf{x})$ that is positive definite and which decreases at each witness state $\mathbf{x}_{j}$ through the control input $\mathbf{u}_{j}$. This function is fed to the verifier, which checks whether $V(\mathbf{x})$ is indeed a CLF, or discover a state $\mathbf{x}_{j+1}$ at which the condition fails. This new state is added to the witness set and the process is iterated.

The procedure described in this paper synthesizes the control Lyapunov function after $60$ iterations of the learning loop and synthesizes the CLF $V(\mathbf{x})$ below:

[TABLE]

This function yields a simple associated control law that can be implemented and guarantees the stabilization of the model (1). Figure 1(b) shows a closed loop trajectory of this control law vs control law extracted by MPC. The advantage of this law is that its calculation is much simpler, and furthermore, the control is formally guaranteed, at least for the model of the system.

Contributions:

In this paper, we instantiate the learning scheme sketched above, and show that under suitable assumptions terminates in finitely many iterations to either yield a control Lyapunov function $V(\mathbf{x})$ that is guaranteed to be valid, or show that Lyapunov function of a specific form does not exist. We demonstrate this scheme and its scalability on several interesting vehicle dynamics taken from the literature to solve stabilization to state and trajectory stabilization problems.

II Background

In this section, we briefly describe control Lyapunov (potential) functions. A state feedback control system $\Psi(X,U,\mathcal{P},\mathcal{C})$ consists of a plant $\mathcal{P}$ and a controller $\mathcal{C}$ over state space $X\subseteq\mathbb{R}^{n}$ and input space $U\subseteq\mathbb{R}^{m}$. The plant $\mathcal{P}$ has a state $\mathbf{x}\in X$ and input $\mathbf{u}\in U$. The vector field for the plant is defined by a smooth function $f:X\times U\rightarrow\mathbb{R}^{n}$. Throughout the paper, we consider control affine systems that are possibly nonlinear in $\mathbf{x}$, but affine in $\mathbf{u}$, of the form

[TABLE]

where $f_{i}:X\rightarrow\mathbb{R}^{n}$. The controller $\mathcal{C}$ measures the state of the plant ($\mathbf{x}\in X$) and provides feedback $\mathbf{u}\in U$. The controller is defined by a continuous feedback function $\mathcal{K}:X\rightarrow U$.

For a given feedback function $\mathcal{K}$, the execution trace of the system $\Psi$ is defined as $\mathbf{x}(.):\mathbb{R}^{+}\rightarrow X$, which maps time to state. Formally, given $\mathbf{x}(0)=\mathbf{x}_{0}$, $\mathbf{x}(.)$ is defined according to $\accentset{\mbox{\large.}}{\mathbf{x}}(t)=f_{0}(\mathbf{x}(t))+\sum_{i=1}^{m}f_{i}(\mathbf{x}(t))(\mathcal{K}(\mathbf{x}(t)))_{i}$, where $\accentset{\mbox{\large.}}{\mathbf{x}}(.)$ is the right derivative of $\mathbf{x}(.)$.

In this article, we study stabilization of nonlinear systems using Lyapunov functions (or potential function) inside a compact set $S$. More precisely, we consider a compact and connected set $S\subset X$. Without loss of generality, the origin $\mathbf{0}$ is the state we seek to stabilize to. Furthermore, $\mathbf{0}\in int(S)$. We restrict the set $S$ to be a basic semi-algebraic set defined by a conjunction of polynomial inequalities. Likewise, the control inputs $U$ are restricted to a polytope.

Our approach relies on control Lyapunov functions. A control Lyapunov function (CLF)  [3] $V$, is a continuous function that respects the following conditions:

[TABLE]

The last condition ensures that the value of $V$ can be decreased everywhere by choosing a proper feedback $u$. Let

[TABLE]

Let $\beta^{*}$ be maximum $\beta$ s.t. $V^{\leq\beta}\subseteq S$. Once a CLF is obtained, it guarantees that the system initialized to any state belonging to $V^{<\beta^{*}}$, can be stabilized to the origin (Fig. 3). A control Lyapunov function guarantees that there is a control strategy, which stabilizes the system.

Theorem 1

Given a control affine system $\Psi$, where $U:\mathbb{R}^{m}$ and a polynomial control Lyapunov function $V$ satisfying Eq. (3), there is a feedback function $\mathcal{K}$ for which if $\mathbf{x}_{0}\in V^{<\beta^{*}}$, then:

$(\forall t\geq 0)\ \mathbf{x}(t)\in S$** 2. 2.

$(\forall\epsilon>0)\ (\exists T\geq 0)\ \|\mathbf{x}(T)-\mathbf{0}\|<\epsilon$* .*

Proof:

1. Using results from Artstein [3], there exists a feedback function $\mathcal{K}^{*}$ s.t. while $\mathbf{x}\in S$, then $\frac{dV}{dt}=\nabla V\cdot f(\mathbf{x},\mathbf{u})<0$. Assuming $\mathbf{x}(0)=\mathbf{x}_{0}\in V^{<\beta^{*}}\subset S$, then initially $V(\mathbf{x}(0))<\beta^{*}$. Now, assume the state reaches $\partial S$ at time $t_{2}$. By continuity of $V$, there is a time $0<t_{1}\leq t_{2}$ s.t. $V(\mathbf{x}(t_{1}))=\beta^{*}$ and $(\forall t\in[0,t_{1}])\ \mathbf{x}(t)\in S$. Therefore

[TABLE]

This means $V(\mathbf{x}(t_{1}))<\beta^{*}$, which is a contradiction. Therefore, the state never reaches $\partial S$ and remains in $int(S)$ forever. 2) $V$ would be a Lyapunov function for the closed loop system when the control unit is replaced with the feedback function $\mathcal{K}^{*}$ and using standard results in Lyapunov theory $(\forall\epsilon>0)\ (\exists T\geq 0)\ ||\mathbf{x}(T)-0||<\epsilon$. ∎

Given, a control Lyapunov function, it is possible to then obtain a feedback law in a closed form that stabilizes the system. Sontag [48] provides a method for extracting a continuous feedback function $\mathcal{K}$ for control affine systems from a control Lyapunov function. This can be extended to systems with constraints on the control inputs [33]. Also, feedback synthesis for periodic time/event triggered switching is possible [43, 7].

We have thus far considered the problem of stabilizing to a fixed equilibrium state. However, given this primitive, we can extend the CLF approach to related problems of (a) Reach-while-stay: reaching a given target set of states $T$ starting from an initial set $I$, while staying inside a safe set $S$ using Lyapunov-barrier functions [39, 44]; (b) Trajectory stabilization: stabilizing to a trajectory $\mathbf{x}(t)$ rather than to a fixed state using time-varying Lyapunov functions; or similarly, (c) Feedback motion planning which addresses the robustness of a plan using funnels [5, 52] ; (d) Obstacles: problems that involve reaching while avoiding an obstacle region in the state-space using artificial potential functions [30]. We will focus our exposition on the basic formulation for stability (Eq. (3)) while demonstrating extensions to some of other applications mentioned above through numerical examples.

III Algorithmic Learning Framework

Finding CLFs is known to be a hard problem, requiring the solution to BMIs [50] or hard polynomial constraints[43]. A standard approach to discovering such functions is to choose a set of basis functions $g_{1},\ldots,g_{r}$ and search of a function of the form

[TABLE]

where $\mathbf{c}\in\mathbb{R}^{r}$ is vector of unknowns. One possible choice of basis functions involves monomials $g_{j}(\mathbf{x}):\ \mathbf{x}^{\alpha_{j}}$ wherein $|\alpha_{j}|_{1}\leq D_{L}$ for some degree bound $D_{L}$ for the learning concept (CLF). Then, the problem is reduced to finding $\mathbf{c}$ s.t. $V_{\mathbf{c}}$ satisfies Eq. (3).

We now present the algorithmic learning framework. Let us fix a control affine system $\mathcal{P}$ over a state-space $X$, control inputs $U$ given by (2). Let $\mathbf{x}^{*}=\mathbf{0}$ be the equilibrium we wish to stabilize the system to, while remaining inside $S\subset X$.

Next, we assume a demonstrator as a function $\mathcal{D}:S\mapsto U$ that given a state $\mathbf{x}\in S$, provides us an appropriate feedback $\mathcal{D}(\mathbf{x})\in U$ for the state $\mathbf{x}$, such that $\mathcal{D}$ is presumed to be a valid function that stabilizes the system.

Remark 1

Our definition of a demonstrator is general enough to allow offline MPC, sample based methods [27, 23], human operator demonstrations [21], or even demonstrations that rely on opaque models such as neural networks.

Also, we assume that the demonstrator is presumed correct. However, the approach can work even if the demonstrator may fail on some input states. Finally, a faulty demonstrator may, at the worst, lead our technique to fail without finding a CLF. In particular, such a demonstrator will not cause our technique to synthesize an incorrect CLF.

Definition 1** (Problem Statement)**

The CLF learning problem has the following inputs:

A dynamical system $\mathcal{P}$ in the form  (2), 2. 2.

A safe set $S$, 3. 3.

A “black-box” demonstrator function $\mathcal{D}:S\mapsto U$ that presumably stabilizes the system, and 4. 4.

A candidate space for CLFs of the form $\sum_{j=1}^{r}c_{j}g_{j}(\mathbf{x})$ given by basis functions $\mathbf{g}(\mathbf{x}):\ \left\langle g_{1}(\mathbf{x}),\ldots,g_{r}(\mathbf{x})\right\rangle$ and a compact set $C\ni(c_{1},\ldots,c_{r})$. We represent the coefficients $(c_{1},\ldots,c_{r})$ collectively as $\mathbf{c}$.

The output can be Success: a function $V_{c}(\mathbf{x}):\mathbf{c}^{t}\cdot\mathbf{g}(\mathbf{x})$ that is a CLF; or Failure: no function could be discovered by our procedure.

III-A Algorithmic Learning Framework

The algorithmic learning framework is shown in Fig. 4, and implements two modules (a) Learner and (b) Verifier that interact with each other and the demonstrator. The framework works iteratively until termination. At the $j^{th}$ iteration, the learner maintains a (witness) set

[TABLE]

$W_{j}$ is a finite set of pairs of states $\mathbf{x}_{i}$ and corresponding demonstrated feedback $\mathbf{u}_{i}$. Corresponding to $W_{j}$, $C_{j}\subseteq C$ is defined as set of candidate coefficients for functions $V_{\mathbf{c}}(\mathbf{x}):\mathbf{c}^{t}\mathbf{g}(\mathbf{x})$ with $\mathbf{c}\in C_{j}$. Formally, $C_{j}$ is a set of all candidates $\mathbf{c}$ s.t. $V_{\mathbf{c}}$ satisfies the CLF condition (3) for every point in the finite set $W_{j}$:

[TABLE]

The flowchart for the overall procedure is shown in Fig. 4.

To begin with, $W_{0}:\emptyset$ and $C_{0}:C$. Each iteration works as follows:

The learner samples a value $\mathbf{c}_{j}\in C_{j}$ and outputs the corresponding function $V_{j}(\mathbf{x}):\mathbf{c}_{j}^{t}\cdot\mathbf{g}(\mathbf{x})$. If $C_{j}=\emptyset$ then no sample is found and the algorithm fails. 2. 2.

The verifier checks if $V_{j}$ is a CLF by checking the conditions in  (3). If $V_{j}$ satisfies the conditions, then the algorithm stops to declare success. Otherwise the verifier selects a (counterexample) state $\mathbf{x}_{j+1}\in S$ for which the CLF condition fails. Assume without loss of generality that $\mathbf{x}_{j+1}\not=\mathbf{0}$. 3. 3.

Failing verification, the demonstrator is called to choose a suitable control $\mathbf{u}_{j+1}$ corresponding to $\mathbf{x}_{j+1}$. 4. 4.

The new set $W_{j+1}:=W_{j}\cup\{(\mathbf{x}_{j+1},\mathbf{u}_{j+1})\}$. Furthermore,

[TABLE]

We now prove some core properties that guarantee the correctness of the proposed scheme. We assume that the learner and the verifier are implemented without any approximations/relaxations (as will be subsequently presented).

Theorem 2

The algorithmic learning framework as described above has the following property:

$\mathbf{c}_{j}\not\in C_{j+1}$. I.e, the candidate found at the $j^{th}$ step is eliminated from further consideration. 2. 2.

If the algorithm succeeds at iteration $j$, then the output function $V_{j}(\mathbf{x})$ is a valid CLF for stabilization. 3. 3.

The algorithm declares failure at iteration $j$ if and only if no linear combination of the basis functions is a CLF compatible with the demonstrator.

Proof:

1. Suppose that $\mathbf{c}_{j}\in C_{j+1}$. Then, $\mathbf{c}_{j}$ satisfies the following conditions (Eq. (6)):

[TABLE]

However, the verifier guarantees that $\mathbf{c}_{j}$ is a counterexample for Eq. (3)). I.e.

[TABLE]

which is a contradiction. Therefore, $\mathbf{c}_{j}\not\in C_{j+1}$.

2. The algorithm declares success if the verifier could not find a counterexample. In other words, $V_{j}$ satisfies conditions of Eq. (3) and therefore a CLF.

3. A CLF $V$ is compatible with a Demonstrator $\mathcal{D}$ iff

[TABLE]

The algorithm declares failure if $C_{j}=\emptyset$ and Eq. (5) is equivalent to $C_{j}:C^{1}_{j}\cap C^{2}_{j}$ where

[TABLE]

Suppose there exists a $\mathbf{c}\in C_{0}$ (a linear combination of the basis function) that yields a CLF $V_{\mathbf{c}}$. If $V_{\mathbf{c}}$ is compatible with $\mathcal{D}$, then

[TABLE]

Therefore, $\mathbf{c}\in C^{1}_{j}$. But we know that $\mathbf{c}\not\in C_{j}=\emptyset$. As a result $\mathbf{c}\not\in C^{2}_{j}$. I.e.

[TABLE]

and it is concluded that $V_{\mathbf{c}}$ is not a CLF, which is a contradiction. ∎

Inverse results [38] suggest polynomial basis for Lyapunov functions are expressive enough for verification of exponentially stable, smooth nonlinear systems. This, justifies using polynomial basis for CLF.

Lemma III.1

Assuming (i) the demonstrator function $\mathcal{D}$ is smooth, (ii) the closed loop system with controller $\mathcal{D}$ is exponentially stable, then there exists a polynomial CLF, compatible with $\mathcal{D}$.

Proof:

Under assumption (i) and (ii), one can show that a polynomial Lyapunov function $V$ (not control Lyapunov function) exists for the closed loop system $\Psi(X,U,\mathcal{P},\mathcal{D})$ [38]. I.e.

[TABLE]

$V$ is also a CLF since it satisfies Eq. (3). Moreover, $V$ is compatible with $\mathcal{D}$ (Eq. (7)). ∎

Lemma III.1 guarantees success of the learning procedure if the set of basis functions is rich enough. We now present implementations of each of the modules involved, starting with the learner.

III-B Learner: Finding a Candidate

The findCandidate function simply samples a point from the set $C_{j}$ defined in Eq. (5). Note that $V_{\mathbf{c}}(\mathbf{x}_{j}):\mathbf{c}^{t}\cdot\mathbf{g}(\mathbf{x}_{j})$ is linear in $\mathbf{c}$ and therefore $\nabla V_{\mathbf{c}}.f(\mathbf{x}_{j},\mathbf{u}_{j})$ is linear in $\mathbf{c}$ as well. The initial space of all candidates $C$ is assumed to be a hyper-rectangular open box. At each iteration, the candidate $\mathbf{c}_{j}\in C_{j}$ is chosen. Suppose the algorithm does not terminate at this iteration. According to Eq. (6), the new set $C_{j+1}$ is obtained as $C_{j+1}:\ C_{j}\cap H_{j}$, wherein $H_{j}$ is defined by two linear inequalities $H_{j1}\land H_{j2}$ ($H_{j1}:\mathbf{a}_{j1}^{t}\mathbf{c}<b_{j1}$, $H_{j2}:\mathbf{a}_{j2}^{t}\mathbf{c}<b_{j2}$). Let $\overline{C_{j}}$ represent the topological closure of the set $C_{j}$.

Lemma III.2

For each $j\geq 0$, $\overline{C_{j}}$ is a convex polyhedron.

Proof:

We prove by induction. Initially $C_{0}$ is an open box and as a result $\overline{C_{0}}$ is a convex polyhedron. Now, assume $\overline{C_{j}}$ is a polyhedron. Recall that $C_{j+1}$ is defined as

[TABLE]

Notice that $f$ and $g_{i}$ are fixed $(\forall i)$ and $\mathbf{x}_{j+1}$ and $\mathbf{u}_{j+1}$ are given. Therefore, $C_{j+1}$ is intersection of an open polyhedron ($C_{j}$) and two half-space which yields an other polyhedron. And since the inequalities in equation above are strict, $C_{j+1}$ is an open polyhedron. ∎

Thus the problem of implementing findCandidate is that of checking emptiness of a polyhedron with some strict inequality constraints. This is readily solved using a slight modification of standard linear programming algorithms using infinitesimals for strict inequalities.

Lemma III.3

There exists a halfspace $H_{jk}:\ \mathbf{a}_{jk}^{t}\mathbf{c}_{j}\geq b_{jk}$ that passes through $\mathbf{c}_{j}$, and $C_{j+1}\subseteq C_{j}\cap H_{jk}$.

By definition $\mathbf{c}_{j}\in C_{j}$. By Theorem 2, it is guaranteed that $\mathbf{c}_{j}\not\in C_{j+1}$. Since $C_{j+1}:C_{j}\cap\{\mathbf{c}|a_{j1}^{t}\mathbf{c}_{j}<b_{j1}\land a_{j2}^{t}\mathbf{c}_{j}<b_{j2}\}$, we can conclude $\mathbf{c}_{j}\not\in\{\mathbf{c}|a_{j1}^{t}\mathbf{c}_{j}<b_{j1}\land a_{j2}^{t}\mathbf{c}_{j}<b_{j2}\}$, I.e.$a_{j1}^{t}\mathbf{c}_{j}\geq b_{j1}\lor a_{j2}^{t}\mathbf{c}_{j}\geq b_{j2}$. Thus one of the two disjuncts holds for $k\in{1,2}$. Further, we note that $C_{j+1}\subseteq C_{j}\cap\{\mathbf{c}|a_{jk}^{t}\mathbf{c}_{j}<b_{jk}\}$.

Choosing a Candidate $\mathbf{c}_{j}$:

The choice of $\mathbf{c}_{j}\in C_{j}$ governs the overall convergence of the algorithm. Figure 5 demonstrates the importance of this choice by showing the candidate $\mathbf{c}_{j}$, the hyperplane $H_{jk}$ and the new region $C_{j+1}$.

We wish to choose $\mathbf{c}_{j}$ s.t. $\mbox{\sc Vol}\left(C_{j+1}\right)\leq\alpha\mbox{\sc Vol}\left(C_{j}\right)$, for a fixed $\alpha$, independent of $H_{jk}$.

Theorem 3** (Tarasov et al.[51])**

Let $\mathbf{c}_{j}$ be chosen as the center of the maximum volume ellipsoid (MVE) of $C_{j}$. Then,

[TABLE]

This leads us to a scheme that guarantees termination of the overall iterative scheme in finitely many steps under a robustness assumption.

Definition 2** (Robust Candidate)**

A candidate $\mathbf{c}\in C$ is $\delta$-robust ($\delta>0$), iff for each $\hat{\mathbf{c}}\in\mathcal{B}_{\delta}(\mathbf{c})$, $V_{\hat{\mathbf{c}}}:\hat{\mathbf{c}}^{t}\cdot\mathbf{g}(\mathbf{x})$ is a CLF, where $\mathcal{B}_{\delta}(\mathbf{c})$ is a ball of radius $\delta$ around $\mathbf{c}$.

Let $\mbox{\sc Vol}(C)<\gamma\Delta^{r}$ for $\gamma>0$ the volume of the unit $r$-ball, and $\Delta>0$. Furthermore, the procedure terminates whenever $\mbox{\sc Vol}(C_{j})<\gamma\delta^{r}$ following the robustness assumption above.

Theorem 4

If at each step $\mathbf{c}_{j}$ is chosen as the center of the maximum volume ellipsoid in $C_{j}$, the learning loop terminates in at most

[TABLE]

Proof:

Initially, $\mbox{\sc Vol}(C_{0})<\gamma\Delta^{r}$. Then by Theorem 3

[TABLE]

Suppose the algorithm does not terminate after $\frac{r(\log(\Delta)-\log(\delta))}{-\log(1-\frac{1}{r})}$ iterations. Then

[TABLE]

and

[TABLE]

And it is concluded that $\mbox{\sc Vol}(C_{j})<\gamma\delta^{r}$, which is the termination condition. This is a contradiction and therefore, the algorithm terminates in at most $\frac{r(\log(\Delta)-\log(\delta))}{-\log(1-\frac{1}{r})}$ iterations. And asymptotically $-\log(1-\frac{1}{r})$ is $\Omega(\frac{1}{r})$ (can be shown using Taylor expansion as $r\rightarrow\infty$) and therefore, the maximum number of iterations would be $O(r^{2})$. ∎

The maximum volume ellipsoid itself can be computed by solving a convex optimization problem[54].

III-C Implementing the Verifier

The verifier given a candidate $V_{j}(\mathbf{x}):\mathbf{c}_{j}^{t}\cdot\mathbf{g}(\mathbf{x})$ checks the CLF conditions in Eq. (3), split into two separate checks:

(A) Check if $V_{j}(\mathbf{x})$ is a positive definite polynomial over $\mathbf{x}\in S$. Assuming that $\mathbf{g}(\mathbf{0})=0$ for the basis, this reduces to:

[TABLE]

(B) Check if the Lie derivative of $V_{j}$ can be made negative for each $\mathbf{x}\in S$ by a choice $\mathbf{u}\in U$:

[TABLE]

This problem seems harder due to the presence of a quantifier alternation. Let $U$ be a polyhedral set defined by $U:\ \{\mathbf{u}\in\mathbb{R}^{m}\ |\ A\mathbf{u}\geq\mathbf{b}\}$. Recall that $f(\mathbf{x},\mathbf{u})$ is control affine function $f_{0}(\mathbf{x})+\sum_{i=1}^{m}f_{i}(\mathbf{x})u_{i}$.

Lemma III.4

Eq. (9) holds for some $\mathbf{x}\in S$ iff

[TABLE]

Proof:

Considering a fixed $V$ and $\mathbf{x}$, then

[TABLE]

which is equivalent to:

[TABLE]

This yields a set of linear inequalities (w.r.t. $\mathbf{u}$). Using Farkas lemma, it is concluded

[TABLE]

∎

In other words, assuming that the dynamics and chosen bases are polynomials, the verification problem reduces to checking if a given semi-algebraic set defined by polynomial inequalities has a solution.

Failing the polynomial assumption, the problem of verification is in general undecidable. However, it can be approximated by techniques such as $\delta$-decision procedures proposed by Gao et al [10]. Solvers like dReal can thus be directly used for the verification problem. While dReal does a good job in adaptive space decomposition, in our experience, they do not scale reliably. Nevertheless, these solvers allow us to conveniently implement a verifier for small but hard problems involving rational and trigonometric functions.

The verification problem for polynomial systems and polynomial CLFs through a polynomial basis function $\mathbf{g}(\mathbf{x})$ is NP-hard, in general. Exact approaches using semi-algebraic geometry and the branch-and-bound solvers (including the dReal approach cited above) can tackle this problem precisely. However, scalability is still an issue.

We sketch a relaxation using SDP solvers [35]. Let us fix a basis of monomial terms of degree up to $D_{V}$, $\mathbf{m}^{t}:[1\ x_{1}\ \ldots\ \mathbf{x}^{D_{V}}]$, wherein $D_{V}$ is chosen as at least half of the maximum degree in $\mathbf{x}$ among all monomials in $g_{j}(\mathbf{x})$ and $\nabla g_{j}\cdot f(\mathbf{x})$: $D_{V}\geq\frac{1}{2}\max\left(\bigcup_{j}\left(\{\mbox{deg}(g_{j})\}\cup\{\mbox{deg}(\nabla g_{j}\cdot f)\}\right)\right).$ Let $Z(\mathbf{x}):\mathbf{m}\mathbf{m}^{t}$. Thus, each polynomial of degree up to $2D_{V}$ may now be written as a trace inner product $p(\mathbf{x})=\ \left\langle P,Z(\mathbf{x})\right\rangle=\mathsf{trace}(PZ(\mathbf{x}))$, wherein the matrix $P$ is constant.

Let $S$ be the semi-algebraic set defined as

[TABLE]

for polynomials $r_{1},\ldots,r_{k}$. The constraint in  (8) is equivalent to solving the following optimization problem over $\mathbf{x}$

[TABLE]

where $V_{j}(\mathbf{x})$ ($r_{i}(\mathbf{x})$) is written as $\left\langle\mathcal{V}_{j},Z(\mathbf{x})\right\rangle$ ($\left\langle R_{i},Z(\mathbf{x})\right\rangle$). and those in  (10) are written as

[TABLE]

wherein the components $\nabla V_{j}\cdot f_{i}(\mathbf{x})$ defining the Lie derivatives of $V_{j}$ are now written in terms of $Z(\mathbf{x})$ as $\left\langle F_{ji},Z(\mathbf{x})\right\rangle$.

The SDP relaxation is used to solve these problems and provide an upper bound of the solution [14]. The result of the relaxation treats $Z(\mathbf{x})$ as a matrix variable $Z$ that will satisfy $Z\succeq 0$. Notice that each optimization problem is feasible simply by setting $Z$ and $\mathbf{\lambda}$ to be zero. However, if the optimal solution of both problems is $Z=0$ in the SDP relaxation, then we will conclude that the given candidate is a CLF.

Lemma III.5

If the relaxed optimization problems in Eqs. (12) and  (13) yield a zero solution, then the given candidate $V_{j}(\mathbf{x})$ is in fact a CLF.

Proof:

Suppose that $V_{j}$ is not a CLF. Then it does satisfies either Eq. (8) or Eq. (9). I.e. $(\exists\mathbf{x}^{*}\neq\mathbf{0})$ s.t. $V_{j}(\mathbf{x}^{*})\leq 0$ or $(\exists\ \mathbf{\lambda}\geq\mathbf{0})\ A_{i}^{t}\mathbf{\lambda}=\nabla V_{j}.f_{i}(\mathbf{x}^{*})(i\in\{1\ldots m\}),\mathbf{\lambda}^{t}\mathbf{b}\geq-\nabla V_{j}.f_{0}(\mathbf{x}^{*})$. Then $Z(\mathbf{x}^{*})$ is a non-zero solution for Eq. (12) or Eq. (13). $Z(\mathbf{x}^{*})$ can also yield a non-zero relaxed matrix $Z^{*}$ which satisfies the relaxed optimization problem.

On the other hand, For any non-zero $Z\succeq 0$, $\left\langle I,Z\right\rangle$ is positive. Therefore, since the optimization returns a zero as the solution, it means the relaxed optimization does not have a non-zero solution (since $\left\langle I,Z\right\rangle$ is being maximized). Which is a contradiction as it is shown non-zero solution exists when $V_{j}$ is not a CLF. Therefore, $V_{j}$ is a CLF. ∎

However, the converse is not true. It is possible for $Z\succ 0$ to be optimal for either relaxed condition, but no $\mathbf{x}\in\mathbb{R}^{n}$ corresponds to the solution. This happens because the relaxation drops two key constraints to convexify the conditions: (1) $Z$ has to be a rank one matrix written as $Z:\mathbf{m}\mathbf{m}^{t}$ and (2) there is a $\mathbf{x}\in\mathbb{R}^{n}$ such that $\mathbf{m}$ is the matrix of monomials corresponding to $\mathbf{x}$.

To deal with this, we adapt our learning framework to work with witnesses $W_{j}:\{(Z_{i},\mathbf{u}_{i})\}_{i=1}^{j}$ replacing states $\mathbf{x}_{i}$ by matrices $Z_{i}$.

Each basis function $g_{j}(\mathbf{x})$ in $\mathbf{g}$ is now written instead as $\left\langle G_{j},Z\right\rangle$. The candidates are therefore, $\sum_{j=1}^{r}c_{j}\left\langle G_{j},Z\right\rangle$. Likewise, we write the components of its Lie derivative $\nabla g_{j}\cdot f_{i}$ in terms of $Z$. 2. 2.

The learner maintains the set $W$ as $\{(Z_{j},\mathbf{u}_{j})\}$, wherein $Z_{j}$ is the feasible solution returned by the SDP solver while solving Eqs. (13) and  (12). In other words, the CLF conditions are, in fact, taken to be the relaxed conditions. 3. 3.

We use a suitable projection operator $\pi$ mapping each $Z$ to a state $\mathbf{x}:\pi(Z)$, such that the demonstrator receives $\mathbf{x}$. In practice, since the vector of monomials used to define $Z$ from $\mathbf{x}$ includes the terms $1,x_{1},\ldots,x_{n}$, the projection operator simply selects a few entries from $Z$ corresponding to the variables. Other more sophisticated projections are also possible.

The relaxed framework thus lifts counterexamples to work over matrices $Z_{j}$. However, the candidate space begins with $C$ and is refined each step as before. I.e, the relaxed framework continues to satisfy Lemmas III.2, III.3, Theorem 4 and Theorem 2 with the definition of (control) Lyapunov function changed to relaxed conditions.

IV Experiments

In this section, we describe numerical results on some example benchmark systems. The algorithmic framework is implemented using quadratic template forms for the CLFs with the tool Globoptipoly used to implement the verifier [13]. The demonstrator is implemented using a nonlinear MPC implemented using a gradient descent algorithm. For each benchmark, we tuned the time horizon, discretization step and the cost function until the control objectives were satisfied by the MPC over hundreds of simulations starting from randomly selected initial states.

Most of the benchmarks consider a reach-while-stay problem, wherein the goal is to reach target set $T$, starting initial set $I$, while remaining in the safe set $S$. We also illustrate an example involving a trajectory stabilization problem. All the computations are performed on a Mac Book Pro with 2.9 GHz Intel Core i7 processor and 16GB of RAM. The reported CLFs are rounded to $2$ decimal points. The summary of results is provided in Table. I.

Bicycle Model:

This system is two-wheeled mobile robot modeled with five states $[x,y,v,\theta,\gamma]$ and two control inputs [9]. The goal is to stabilize the car to a target velocity $v^{*}=5$, as shown in Fig. 6. We drop the variable $x$ (since it is immaterial to our stabilization problem) and obtain a model with four state variables:

[TABLE]

where $\sigma=tan(\gamma)$ (see Fig. 6). $\sin$ function is approximated with a polynomial of degree $1$. The method finds the following CLF:

[TABLE]

Inverted Pendulum on a Cart:

This example has applications in balancing two-wheeled robots [6] (cf. [1] for list of such robots). The system has four state variables $[x,\accentset{\mbox{\large.}}{x},\theta,\accentset{\mbox{\large.}}{\theta}]$ and one input $u$. The dynamics, after partial linearization, have the following form [24]:

[TABLE]

where $m=0.21$, $M=0.815$, $g=9.8$ and $l=0.305$. The trigonometric and rational functions are approximated with polynomials of degree $3$. The sets are $S:[-1,1]^{4},U:[-20,20],I:\mathcal{B}_{0.1}(\mathbf{x}),T:\mathcal{B}_{0.1}(\mathbf{x})$. We obtain the following CLF:

[TABLE]

Forward Flight for Caltech Ducted Fan:

The Caltech ducted fan models the aerodynamics of a single wing of a thrust vectored, fixed wing aircraft [16]. This problem is to design forward flight control in which the angle of attack needs to be changed. The model of the system is carefully calibrated through wind tunnel experiments. The system has $4$ states $[v,\gamma,\theta,q]$ and two control inputs: $u$ and $\delta_{u}$. The dynamics are:

[TABLE]

where $\alpha=\theta-\gamma$, and $D$, $L$, and $M$ are polynomials in $v$ and $\alpha$. For full list of parameters, see [16]. The goal is to reach $\mathbf{x}^{*}:\ [6,0,0.1771,0]$ as the steady state. We perform a translation so that the $\mathbf{x}^{*}$ is the origin in the new coordinate system.

[TABLE]

First, we approximate $v^{-1}$, $\sin$ and $\cos$ with polynomials of degree $1$, $3$ and $3$ respectively, to get polynomial dynamics. However, the system is not affine in control. We replace $u$ and $\delta_{u}$ input variables with $u_{1}=u\sin(\delta_{u})$ and $u_{2}=u\cos(\delta_{u})$ and $U$ is under-approximated with a polytope. These changes yield a polynomial control affine dynamics, which fits the description of our model. The procedure finds the following CLF:

[TABLE]

Hover Mode for Caltech Ducted Fan:

This problem is again taken from [16]. The goal is to keep the ducted fan in a hover mode. The system has $6$ variables $x$, $y$, $\theta$, $\accentset{\mbox{\large.}}{x}$, $\accentset{\mbox{\large.}}{y}$, $\accentset{\mbox{\large.}}{\theta}$ and two control inputs $u_{1}$, $u_{2}$. The dynamics are

[TABLE]

where $m=11.2$, $g=0.28$, $J=0.0462$, $r=0.156$ and $d_{c}=0.1$. The sets are:

[TABLE]

The $sin$ and $cos$ are approximated with polynomials of degree $2$ and the procedure finds a quadratic CLF:

[TABLE]

Unicycle:

Now, we consider a simpler system, namely the unicycle model [28], which is known not to have a polynomial CLF. However, for trajectory tracking problem, one can provide a time varying CLF [52] for a finite time horizon control (using funnels). The unicycle model has the dynamics: $\accentset{\mbox{\large.}}{x_{1}}=u_{1},\accentset{\mbox{\large.}}{x_{2}}=u_{2},\accentset{\mbox{\large.}}{x_{3}}=x_{1}u_{2}-x_{2}u_{1}$.

We consider a planning problem: starting near $[\frac{\pi}{2},-1,-1]$, the goal is to reach near $[0,2,0]$, and by near we mean within distance $1$. In the first step, a feasible trajectory $\mathbf{x}(t)$ is generated as shown in Fig. 7. Then $\mathbf{x}(t)$ is approximated with piecewise polynomials. More precisely, trajectory consists of two segments. The first segment brings the car to the origin and the second segment moves the car to the destination. Each segment is approximated using polynomials in $t$ with degree up to $3$. The time varying CLF $V$ is a function of both the state $\mathbf{x}$ and time $t$. Choosing a template for $V$, our method is applied to this problem and we are able to find a strategy to implement the plan with guarantees. A level-set of the funnel (CLF) is shown in Fig. 7.

V Related Work

Synthesis of Lyapunov Functions from Data:

The problem of synthesizing Lyapunov functions for a control system by observing the states of the system in simulation has been investigated in the past by Topcu et al to learn Lyapunov functions along with the resulting basin of attraction [53]. Whereas the original problem is bilinear, the use of simulation data makes it easier to postulate states that belong to the region of attraction and therefore find Lyapunov functions that belong to this region by solving LMIs in each case. The application of this idea to larger black-box systems is demonstrated by Kapinski et al [20]. Our approach focuses on controller synthesis through learning a control Lyapunov function in a bid to replace an existing controller. In doing so, we do not attempt to prove that the original demonstrator is necessarily correct but find a control Lyapunov function by assuming that the demonstrator is able to stabilize the system for the initial conditions we query on. Another important contribution lies in our analysis of the convergence of the learning with a bound on the maximum number of queries needed. In fact, these results can also be applied to the Lyapunov function synthesis approaches mentioned earlier.

Counter-Example Guided Inductive Synthesis (CEGIS):

Our approach of alternating between a learning module that proposes a candidate and a verification module that checks the proposed candidate is identical to the CEGIS framework originally proposed by Solar-Lezama et al. [47, 46]. As such, the CEGIS approach does not include a demonstrator that can be queried. The extension of this approach Oracle-guided inductive synthesis [18], generalizes CEGIS using an oracle that serves a similar role as a demonstrator in this paper. Jha et al. prove bounds on the number of queries for discrete concept classes using results on exact concept learning in discrete spaces [11].

The CEGIS procedure has been used for the synthesis of CLFs recently by authors [43, 44], combining it with SDP solvers for verifying CLFs and a robust version for switched systems. The key difference here lies in the use of the demonstrator module that simplifies the learning module. In the absence of a demonstrator module, the problem of finding a candidate reduces to solving linear constraints with disjunctions, an NP-hard problem [44]. Likewise, the convergence results are quite weak [43]. In the setting of this paper, however, the use of a MPC scheme as a demonstrator allows us to use faster LP solvers and provide convergence guarantees.

Learning from Demonstration:

The idea of learning from demonstration has had a long history in robotics [2]. The overall framework uses a demonstrator that can in fact be a human operator [21] or a complex MPC-based control law [19, 56, 57, 34]. The approaches differ on the nature of the interaction between the learner and the demonstrator; as well as how the policy is inferred. Our approach stands out in many ways: (a) We represent our policies by CLFs which are polynomial. On one hand, these are much less powerful than approaches that use neural networks [56, 19], for instance. However, the advantage lies in our ability to solve verification problems to ensure that the resulting policy learned through the CLF is correct with respect to the underlying dynamical model. (b) Our framework is adversarial. The choice of the counterexample to query the demonstrator comes from a failed attempt to validate the current candidate. (c) Finally, we use simple yet powerful ideas from convex optimization to place bounds on the number of queries, paralleling some results on concept learning in discrete spaces [11].

Control Lyapunov functions were originally introduced by Artstein and the construction of a feedback law given a CLF was first given by Sontag [3, 48]. As such, the problem of learning CLFs is well known to be hard, involving bilinear matrix inequalities (BMIs) [50]. An equivalent approach involves solving bilinear problems simultaneously for a control law and a Lyapunov function certifying it [8, 31]. BMIs are well known to be NP-hard, and hard to solve for a feasible solution [13]. The common approach is to perform an alternating minimization by fixing one set of bilinear variables while minimizing in the other. Such an approach has poor guarantees in practice, often “getting stuck” on a saddle point that does not allow the technique to make progress in finding a feasible solution. To combat this, Majumdar et al. (ibid) use LQR controllers and their associated Lyapunov functions for the linearization of the dynamics as good initial seed solutions [31]. In contrast, our approach simply assumes a demonstrator in the form of a MPC controller that can be used to resolve the bilinearity. Furthermore, our approach does not encounter the local saddle point problem.

The use of the learning framework with a demonstrator distinguishes the approach in this paper from recently developed ideas based on formal synthesis [55, 29, 45, 36, 37, 22, 49, 43, 44, 15]. These techniques focus on a given dynamical system and a specification of the correctness in temporal logic to solve the problem of controller design to ensure that the resulting trajectories of the closed loop satisfy the temporal specifications. Majority of the approaches are based on discretization of the state-space into cells to compute a discrete abstraction of the overall system [55, 29, 45, 36, 37, 22]. A smaller set of approaches synthesize Lyapunov-like functions by solving nonlinear constraints either through branch-and-bound techniques or by sum-of-square (SOS) relaxation techniques [49, 43, 44].

In this paper, we use the Lyapunov function approach to synthesizing controllers. An alternative is to use occupation measures [42, 40, 26, 32]. These methods formulate an infinite dimensional problem to maximize the region of attraction and obtain a corresponding control law. This is relaxed to a sequence of finite dimensional SDPs [25]. Note however that the approach computes an over approximation of the finite time backward reachable set from the target and a corresponding control. Our framework here instead seeks an under approximation that yields a guaranteed controller.

VI Conclusion And Future Work

We have proposed an algorithmic learning framework for synthesizing CLFs using a demonstrator and demonstrated our approach on challenging numerical examples with 4-8 state variables. As future work, we are considering many directions including the extensions to noisy/erroneous demonstrations, using output feedback (rather than full state feedback) synthesis and allowing disturbances in our framework. We are also working on integrating our control framework with RRT-based path planning and implementing it on board robotic vehicles.

Acknowledgments

This work was funded in part by NSF under award numbers SHF 1527075 and CPS 1646556. All opinions expressed are those of the authors and not necessarily of the NSF.