Learning Dynamic-Objective Policies from a Class of Optimal Trajectories

TL;DR

This paper introduces a new method combining trajectory optimisation, homotopy continuation, and imitation learning to efficiently synthesize dynamic-objective state-feedback controllers capable of switching objectives online.

Contribution

It presents a novel approach to generate and train policies for multiple objectives using numerical continuation and imitation learning, enabling online objective switching.

Findings

Policies produce near-optimal trajectories in simulations.

Method effectively learns families of optimal controllers.

Demonstrated on inverted pendulum and spacecraft transfer.

Abstract

Optimal state-feedback controllers, capable of changing between different objective functions, are advantageous to systems in which unexpected situations may arise. However, synthesising such controllers, even for a single objective, is a demanding process. In this paper, we present a novel and straightforward approach to synthesising these policies through a combination of trajectory optimisation, homotopy continuation, and imitation learning. We use numerical continuation to efficiently generate optimal demonstrations across several objectives and boundary conditions, and use these to train our policies. Additionally, we demonstrate the ability of our policies to effectively learn families of optimal state-feedback controllers, which can be used to change objective functions online. We illustrate this approach across two trajectory optimisation problems, an inverted pendulum swingup and a spacecraft orbit transfer, and show that the synthesised policies, when evaluated in simulation, produce trajectories that are near-optimal. These results indicate the benefit of trajectory optimisation and homotopy continuation to the synthesis of controllers in dynamic-objective contexts.

Tables

Table 1. TABLE I : The final M ​ S ​ E 𝑀 𝑆 𝐸 MSE losses in training ( ϵ italic-ϵ \epsilon ) and validation ( ϵ ′ superscript italic-ϵ ′ \epsilon^{\prime} ) of π 0 subscript 𝜋 0 \pi_{0} and π 1 subscript 𝜋 1 \pi_{1} in the regression of D 0 subscript 𝐷 0 D_{0} and D 1 subscript 𝐷 1 D_{1} , respectively.

$\text{Nodes}\times \text{Layers}$	${ϵ}_0$	${ϵ}_0^{\prime }$	${ϵ}_1$	${ϵ}_1^{\prime }$
$50\times 2$	0.043434	0.049832	0.014710	0.013711
$50\times 4$	0.034315	0.043264	0.014066	0.012860
$100\times 2$	0.038298	0.044142	0.018373	0.018608
$100\times 4$	0.029807	0.036758	0.012590	0.011232

Table 2. (a) Inverted pendulum swingup

	$m\times n$
$\alpha$	$50\times 2$	$50\times 4$	$100\times 2$	$100\times 4$
0.0	8.366217	0.201963	6.911897	0.625839
0.1	2.310301	2.667611	2.722436	2.669494
0.2	1.270388	4.418089	1.152010	3.045441
0.3	3.473601	4.555533	3.798529	5.253961
0.4	4.832105	3.444665	4.429260	4.477721
0.5	3.260343	1.645225	2.438281	2.761684
0.6	1.209908	0.616588	0.729584	1.150946
0.7	0.669214	0.430044	0.279920	0.144534
0.8	0.497213	0.514005	0.356340	2.509390
0.9	0.904339	0.797179	0.485849	0.836367
1.0	2.309510	1.775131	1.027916	1.200968
Mean	2.645740	1.915094	2.212002	2.243304

Table 3. (a) Inverted pendulum swingup

	$m\times n$
$\alpha$	$50\times 2$	$50\times 4$	$100\times 2$	$100\times 4$
0.0	8.366217	0.201963	6.911897	0.625839
0.1	2.310301	2.667611	2.722436	2.669494
0.2	1.270388	4.418089	1.152010	3.045441
0.3	3.473601	4.555533	3.798529	5.253961
0.4	4.832105	3.444665	4.429260	4.477721
0.5	3.260343	1.645225	2.438281	2.761684
0.6	1.209908	0.616588	0.729584	1.150946
0.7	0.669214	0.430044	0.279920	0.144534
0.8	0.497213	0.514005	0.356340	2.509390
0.9	0.904339	0.797179	0.485849	0.836367
1.0	2.309510	1.775131	1.027916	1.200968
Mean	2.645740	1.915094	2.212002	2.243304

Table 4. (b) Spacecraft orbit transfer

	$m\times n$
$\alpha$	$50\times 2$	$50\times 4$	$100\times 2$	$100\times 4$
0.0	5.097919	0.163055	0.264219	0.094422
0.1	3.551325	0.057574	0.109408	0.073708
0.2	2.160594	0.355176	0.468579	0.353486
0.3	0.954027	0.665326	0.754457	0.675646
0.4	0.117137	0.979295	0.975896	1.031487
0.5	0.519775	0.344638	0.242022	0.455965
0.6	0.832056	0.210354	0.353348	0.051248
0.7	0.940100	0.622716	0.765109	0.439597
0.8	1.231413	1.173717	1.259935	1.005639
0.9	0.799833	1.096328	1.055267	1.012649
1.0	1.103462	0.189937	0.768724	0.044418
Mean	1.573422	0.532556	0.637906	0.476206

Equations

$$\mathcal{J}=(1-\alpha ){\int }_{t_0}^{t_f}\mathcal{L}(\mathbf{u}(t))dt+\alpha {\int }_{t_0}^{t_f}{\mathcal{L}}^{\prime }(\mathbf{u}(t))dt,$$

$$\dot{\mathbf{s}}=\left[\begin{array}{c}\dot{x}\\ \dot{v}\\ \dot{\theta }\\ \dot{\omega }\end{array}\right]=\left[\begin{array}{c}v\\ u\\ \omega \\ \sin \left(\theta \right)-u\cos \left(\theta \right)\end{array}\right]=\mathbf{f}(\mathbf{s},u),$$

$$\mathcal{J}=\left(1-\alpha \right){\int }_{t_0}^{t_f}u{\left(t\right)}^{2}dt+\alpha {\int }_{t_0}^{t_f}dt,$$

$$\begin{gathered} \mathcal{H}={\mathbf{\lambda }}^{⊺}\cdot \mathbf{f}(\mathbf{s},u)+\mathcal{L}={\lambda }_{x}v+{\lambda }_{v}u+{\lambda }_{\theta }\omega \\ +{\lambda }_{\omega }\left(\sin \left(\theta \right)-u\cos \left(\theta \right)\right)+\left(1-\alpha \right)u^2+\alpha , \end{gathered}$$

$$u^{\star }=\underset{u}{\mathrm{argmin}}\left(\mathcal{H}\right)=\frac{{\lambda }_v-{\lambda }_{\omega }\cos \left(\theta \right)}{2\left(\alpha -1\right)},$$

$$\sigma ={\lambda }_v-{\lambda }_{\omega }\cos \left(\theta \right),$$

$$\underset{\alpha \to 1}{\lim }\left(u^{\star }\right)=\{\begin{array}{ll}-1& \mathrm{if}\sigma <0\\ 1& \mathrm{if}\sigma >0\end{array},$$

$$\dot{\mathbf{\lambda }}=-{\nabla }_{\mathbf{s}}\mathcal{H}=\left[\begin{array}{c}0\\ -{\lambda }_x\\ -{\lambda }_{\omega }\left(u\sin \left(\theta \right)+\cos \left(\theta \right)\right)\\ -{\lambda }_{\theta }\end{array}\right].$$

$$\dot{\mathbf{s}}=\left[\begin{array}{c}\dot{\mathbf{r}}\\ \dot{\mathbf{v}}\\ \dot{m}\end{array}\right]=\left[\begin{array}{c}\mathbf{v}\\ \frac{Tu}{m}\hat{\mathbf{u}}-\frac{\mu }{r^3}\mathbf{r}\\ -\frac{Tu}{I_{sp}{g}_0}\end{array}\right]=\mathbf{f}(\mathbf{s},\mathbf{u}),$$

$$\mathcal{J}=\left(1-\beta \right){\int }_{t_0}^{t_f}u(t{)}^{2}dt+\beta {\int }_{t_0}^{t_f}\left(\alpha +u(t)\left(1-\alpha \right)\right)dt,$$

$$\begin{gathered} \mathcal{H}={\mathbf{\lambda }}_r\cdot \mathbf{v}+{\mathbf{\lambda }}_v\cdot \left(\frac{Tu}{m}\hat{\mathbf{u}}-\frac{\mu }{r^3}\mathbf{r}\right)+{\lambda }_m\left(-\frac{Tu}{I_{sp}{g}_0}\right) \\ +\beta \left(\alpha +u\left(1-\alpha \right)\right)+u^2\left(1-\beta \right) \end{gathered}$$

$${\hat{\mathbf{u}}}^{\star }=-\frac{{\mathbf{\lambda }}_v}{{\lambda }_v}.$$

$$u^{\star }=\frac{1}{2(1-\beta )}\left(\frac{T\left({\mathbf{\lambda }}_v\cdot {\mathbf{\lambda }}_v\right)}{{\lambda }_{v}m}+\frac{T{\lambda }_m}{I_{sp}{g}_0}+\beta \left(\alpha -1\right)\right),$$

$$\sigma =\frac{{\mathbf{\lambda }}_v\cdot {\mathbf{\lambda }}_v}{m{\lambda }_v}+{\lambda }_m+1-\alpha ,$$

$$\underset{\beta \to 1}{\lim }\left(u^{\star }\right)=\{\begin{array}{ll}0& \mathrm{if}\sigma <0\\ 1& \mathrm{if}\sigma >0\end{array},$$

$$\dot{\mathbf{\lambda }}=\left[\begin{array}{c}\frac{\mu }{r^3}{\mathbf{\lambda }}_v-\frac{3\mu }{r^5}\left({\mathbf{\lambda }}_v\cdot \mathbf{r}\right)\\ -{\mathbf{\lambda }}_r\\ \frac{Tu}{m^2}\left({\mathbf{\lambda }}_v\cdot \hat{\mathbf{u}}\right)\end{array}\right].$$

$$\frac{r^3\left({\mathbf{\lambda }}_v\cdot \mathbf{v}\right)-\mu \left({\mathbf{\lambda }}_r\cdot \mathbf{r}\right)}{\sqrt{{\mu }^2\left(\mathbf{r}\cdot \mathbf{r}\right)+\left(\mathbf{v}\cdot \mathbf{v}\right){\left(\mathbf{r}\cdot \mathbf{r}\right)}^3}}=0,$$

Full text

I Introduction

An optimal controller that is capable of changing its objective function in the middle of a mission can be very useful in an autonomous system. Imagine a vehicle that started off on a time-optimal trajectory towards a given destination but suddenly finds itself having less energy than expected; perhaps due to a battery failure, a broken solar panel, or a leaking fuel tank. It can then instantly switch to an effort-optimal controller. Conversely, a vehicle that was saving energy for future transitions might suddenly be in a hurry to reach a destination and can then instantly switch to a controller that reaches the goal in the shortest possible time.

In general, applying an optimal control policy is desirable, but often infeasible for many autonomous systems. The reason for this is that, for many systems characterised by nonlinear dynamics, finding a closed-loop solution to the underlying optimal control is intractable. A common approach to circumvent this is to precompute an open-loop solution, through trajectory optimisation [1], and apply a simple state-feedback controller to track it. However, such an approach suffers when unforeseen events require a change of the objective function or lead the system’s state out of the controller’s region of attraction.

An alternative approach is to apply model predictive control (MPC), where open-loop solutions are incrementally computed and executed on a finite-time horizon. Although this approach provides reactivity in response to unforeseen events, it might suffer in performance, as it does not consider the full-time horizon; and in computational complexity, as it requires repeatedly solving an optimisation problem.

In recent years, with the progression of machine learning (ML), data-driven approaches have become popular in the context of control. Across these approaches, control policies are represented by neural networks (NN) [2], whose feed-forward architectures express simple executable functions. When given appropriate data, these models can be trained to approximate optimal control policies, and pose a reasonable alternative to MPC in terms of both performance and computational complexity [3], especially on low-end hardware.

Whilst the problem of synthesising optimal control policies has been well addressed in ML literature, in both single and multi-objective contexts [4, 5, 6], relatively few have tackled the problem of adapting policies online to changing objectives [7, 8]. Such a case is particularly relevant to systems in which objective priorities can change unexpectedly, e.g. saving energy vs. saving time. To the best our knowledge, the use of trajectory optimisation and imitation learning has been unexplored in this regard.

In this work, we show that we can straightforwardly leverage trajectory optimisation and homotopy continuation to synthesise state-feedback controllers capable of adapting online to dynamic objectives. Our contributions are:

we develop an efficient data-generation pipeline to produce optimal demonstrations across multiple objectives and boundary conditions; 2. 2.

and we show that our trained policies can continuously represent homotopies between optimal policies.

The outline of the paper is as follows. In Section II we describe related work. Then, in Section III, we explain the homotopy continuation method we use for trajectory optimisation. In Section IV, we explain our method of dataset generation, our machine learning setup, and the results we achieved. Finally, in Section V, we reflect upon our results and discuss what they entail.

II Related Work

Many problems in robotics are naturally formulated in terms of nonlinear optimal control problems [9, 10, 11], but they often lack a closed-form solution. One can usually, however, solve the trajectory optimisation problem (TOP) using numerical methods [1], to obtain open-loop solutions. But, as described before, naïvely executing or tracking these solutions might have drawbacks in terms of robustness.

MPC addresses this issue by iteratively computing and executing an open-loop solution on a finite-time horizon, allowing for online reactions in response to unforeseen events. With advances in MPC’s efficiency [12] and increasing computational capabilities of many robotic platforms, this approach has become popular in a variety of applications [11, 13, 14, 15, 16, 17]. But, although it can be quite effective, the need to employ a finite-time horizon, regardless of computational capabilities, results in suboptimal trajectories, as they do not take the full-time horizon into account.

A set of approaches that mitigates these issues, by learning the optimal control with respect to the full- (or infinite-) time horizon, uses reinforcement learning (RL) [18, 19]. These approaches operate by constructing a policy through incremental interaction with the environment. Although such approaches have been quite successful in the continuous control domain [2], particularly for stochastic systems, they sometimes struggle to converge and are sensitive to the choice of rewards [20, 21].

In the case that expert demonstrations are either readily available or producible, it is sometimes more straightforward to rely on the alternative ML approach: imitation learning (IL). In this approach, expert demonstrations are utilised to synthesise optimal policies in a supervised-learning manner. This approach benefits from a greater sampling-efficiency and has been shown to rapidly accelerate policy-convergence in RL approaches [22, 23]. It is usually the case that demonstrations are not necessarily optimal, but if, however, the system at hand is well-suited to providing optimal demonstrations, e.g. through trajectory optimisation, this approach proves to be particularly useful in converging to an optimal policy in an uncomplicated way [5, 24, 25, 26].

A commonality among these approaches — IL and RL — is that their policies are represented by NNs and are traditionally orientated towards single-objective scenarios only. Some pertaining works have indeed addressed the multi-objective111The single- and multi-objective modalities of trajectory optimisation are often referred to as single- and multi-task in ML parlance, respectively. Whilst, in ML, these terms often describe boundary conditions associated with different tasks, we maintain that an objective describes only the cost of a trajectory ($\mathcal{J}$). case, but typically have only sought policies lying along the Pareto front [6], and only few have targeted the case where objective priorities may change over time [7, 8].

In the virtual-first work to address this case [7], a set of policies was learnt in an RL setting, to discretely represent a spectrum of different objectives, which could then be referred to online to adapt to changing objective priorities. However, as noted in [8], using a discrete set of policies does not scale to complex problems nor truly represent a continuous mapping between objectives. Recently, [8] showed that policies could be adapted online to dynamic preferences in an RL setting, using a Q-network, conditioned on the objective-priority. We diverge from these works in that we show how trajectory optimisation and homotopy can be straightforwardly leveraged to synthesise these conditional policies in an IL setting.

In [5], it was shown that Pontryagin’s minimum principle (PMP)222PMP is a manifestation of indirect trajectory optimisation; alternatively, direct methods could be used to generate optimal demonstrations in the case of a black-box dynamics model. For an overview of trajectory optimisation methods refer to [1]. [27] could be used to generate optimal demonstrations in order to synthesise optimal policies in an IL setting. It was shown, across a variety of two-dimensional dynamical systems, that the resulting policies, when executed in simulation, produced near-optimal trajectories, even when the systems were initialised in unseen regions of the state-space. In [24], we showed that this approach could be extended to three dimensions and used to encode manifold boundary conditions in the state-space, using transversality conditions of PMP. These results were built upon in [25], where an investigation of NN hyperparameters and an improved evaluation metric — policy trajectory optimality — were presented.

In these works, optimal policies were synthesised for a single objective. However, in both [5] and [26], homotopy continuation [28] was employed between different objectives to ease the burden of generating demonstrations under objectives which are particularly challenging for gradient-based optimisers. Such a case arises when the topology of the TOP becomes discontinuous due to the underlying optimal control being bang-bang. We build upon these works by taking advantage of the full homotopy path between objectives to straightforwardly synthesise conditional policies, which can react online to changing objective priorities, as we will describe in more detail in the following sections.

III Trajectory optimisation

In this section, we outline our optimal-trajectory dataset generation process, used for learning in Section IV.

Throughout this paper, we consider nonlinear autonomous dynamical systems of the form $\dot{\boldsymbol{s}}=\boldsymbol{f}(\boldsymbol{s},\boldsymbol{u})$, with scalar objectives of the form $\mathcal{J}=\int_{t_{0}}^{t_{f}}\mathcal{L}(\boldsymbol{u}(t))\operatorname{d\!}t$. With these, one can solve the TOP to determine an optimal trajectory $[\boldsymbol{s}(t),\boldsymbol{u}(t)]^{\intercal}$ — an open-loop solution — given an initial and desired state. In this work, we formulate this problem as a nonlinear programme and solve it using numerical optimisation [1], where we must determine the optimal decision vector $\boldsymbol{z}^{\star}$ to obtain the optimal trajectory. For brevity, we denote the TOPs in Sections III-B and III-C as $P_{0}$ and $P_{1}$, respectively.

III-A Homotopy continuation

Under certain objectives, computing an optimal trajectory can be rather difficult; an example is when the optimal control profile $\boldsymbol{u}^{\star}(t)$ is discontinuous, e.g. in the bang-bang case. In order to decrease the burden of finding a solution, homotopy continuation, or regularisation techniques, can be employed to achieve convergence. This method is often used to solve singular-control problems [28].

To elaborate further: given two different objectives, one can define a linear homotopy between them

[TABLE]

where the homotopy parameter $\alpha\in[0,1]$ characterises the objectives’ priorities. Assuming $\mathcal{L}$ characterises the easier objective, we can solve the TOP first with $\alpha=0$, then use its solution to resolve the same problem with a slightly increased value. With this process, we iteratively solve the TOP until $\alpha=1$, at which point we will have converged to a solution under the difficult objective. Through this process, we obtain a convergent series of solutions — a path-homotopy. If there are several different objectives, one can extend this approach to multiple homotopies, as we do in Section III-C. This process is further described in Algorithm 1, where the boundary conditions remain constant, $\mathtt{solve}(\boldsymbol{s}_{0},\boldsymbol{z},\alpha$) minimises (1) from an initial state $\boldsymbol{s}_{0}$, and $\alpha^{*}$ is the last successful solution parameter.

Similarly, to obtain a database that encompasses a diverse set of states, we can also utilise continuation upon the initial state. To do this, we slightly perturb the initial state and resolve the TOP with the previous solution as the initial guess, again leading to a convergent series of solutions. We further describe this process in Algorithm 2, where the homotopy parameter remains constant, $\delta$ defines the perturbation size, and $n$ is the desired number of solutions. Through these algorithms, we obtain a large set of TOP solutions $\boldsymbol{T}$, that encompass the entire range of homotopy parameters and a broad range of initial states.

III-B Inverted pendulum swing up

For our first test case, we consider the simple nondimensional inverted pendulum [29],

[TABLE]

where the variables are: cart position $x$, cart velocity $v$, pole angle $\theta$, pole angular velocity $\omega$, and control $u\in[-1,1]$.

We begin by defining a homotopic objective

[TABLE]

defined from quadratically optimal ($\alpha=0$) to time-optimal ($\alpha=1$) control. Using PMP, we define the Hamiltonian

[TABLE]

where $\boldsymbol{\lambda}$ is the Lagrange multiplier vector — or costate — having the same dimensionality as the state. We then minimise $\mathcal{H}$ with respect to $u$ to find the optimal control

[TABLE]

defined for $\alpha\in[0,1)$. As we drive the homotopy parameter to unity, we find the optimal switching function

[TABLE]

defining the bounded time-optimal control

[TABLE]

where it should be noted that the edge case $\sigma=0$ is only encountered instantaneously, if at all, so singular control analysis is not necessary here. Lastly, we compute the costate equations

[TABLE]

To encode the swingup task into the TOP, we enforce the equality constraint $\boldsymbol{s}(t_{f})=\boldsymbol{0}$. With the shooting method [1], the decision vector becomes $\boldsymbol{z}=[T,\boldsymbol{\lambda}(t_{0})]^{\intercal}$, where $T=t_{f}-t_{0}$ is the trajectory duration and $\boldsymbol{\lambda}(t_{0})$ is the initial costate. Following Algorithm 1 and considering the nominal downright state $\boldsymbol{s}_{0}=[0~{}0~{}\pi~{}0]^{\intercal}$, we compute our initial path-homotopy between quadratically and time-optimal control, as shown in Figure 1(a).

III-C Spacecraft orbit transfer

As an increase in complexity of boundary conditions and dimensionality, we now consider the problem of optimising a spacecraft’s trajectory from Earth to the orbit of Mars. Its heliocentric dynamics are given by

[TABLE]

where the state variables are: position $\boldsymbol{r}=[x,y,z]^{\intercal}$, velocity $\boldsymbol{v}=[v_{x},v_{y},v_{z}]^{\intercal}$, and mass $m$. The controls are the thrust’s throttle $u\in[0,1]$ and direction $\hat{\boldsymbol{u}}=[\hat{u}_{x},\hat{u}_{y},\hat{u}_{z}]^{\intercal}$. The constant parameters governing the dynamics are: maximum thrust capability $T=0.2~{}[N]$, specific impulse $I_{sp}=2500~{}[s]$, gravitational acceleration at sea level $g_{0}=9.81~{}[m\cdot s^{-1}]$, and standard gravitational parameter of the sun $\mu=1.3271\times 10^{20}~{}[m^{3}\cdot s^{-2}]$.

We again construct a homotopic objective,

[TABLE]

where we have added a secondary homotopy parameter $\beta\in[0,1]$ as a means to feasibly arrive to a solution of the problem at $\alpha\in[0,1],\beta=1$. The homotopies between optimal controls are defined as: quadratic to effort $\alpha=0,\beta\in[0,1]$, quadratic to time $\alpha=1,\beta\in[0,1]$, and effort to time $\alpha\in[0,1],\beta=1$. Using PMP, we then define the Hamiltonian

[TABLE]

To find the optimal thrust direction $\hat{\boldsymbol{u}}^{\star}$, we isolate the portion of $\mathcal{H}$ that depends on it: $\frac{Tu}{m}\left(\hat{\boldsymbol{u}}\cdot\boldsymbol{\lambda}_{v}\right)$. Considering that $T$, $m$, and $u$ are positive, $\hat{\boldsymbol{u}}^{\star}$ must be directed opposite of $\boldsymbol{\lambda}_{v}$ to be a minimiser:

[TABLE]

Substituting $\hat{\boldsymbol{u}}^{\star}$ into $\mathcal{H}$ and minimising it with respect to $u$, we obtain

[TABLE]

defined for $\alpha\in[0,1],\beta\in[0,1)$. Driving $\beta$ to unity, we find

[TABLE]

defining the bounded control

[TABLE]

expressing the homotopy between effort- ($\alpha=0$) and time-optimal control ($\alpha=1$), which are discontinuous. Finally, we compute

[TABLE]

To finish defining the TOP, we enforce two transversality conditions. The first is to allow the optimiser to choose the best final mass $\lambda_{m}(t_{f})=0$, since it is affected by the control. The second is to allow the optimiser to choose the best terminal state along Mars’s orbit

[TABLE]

see [24] for details. The decision vector becomes $\boldsymbol{z}=[T,M(t_{f}),\boldsymbol{\lambda}(t_{0})]^{\intercal}$, where $T=t_{f}-t_{0}$ is the time of flight, $M(t_{f})$ is the mean anomaly of the final orbit, and again $\boldsymbol{\lambda}(t_{0})$ is the initial costate. Following Algorithm 1 again and considering Earth’s position and velocity as the initial state, and $m(t_{0})=1000~{}[kg]$, we compute each homotopy path, as shown in Figures 2(a)–2(c), for an interplanetary transfer to somewhere along Mars’s orbit333We pick the starting time to be $0~{}[MJD2000]$, using Modified Julian Date notation..

IV Learning

In this section we describe the generated datasets in detail, outline the implemented NN architectures, and evaluate the results of the learning process.

IV-A Datasets

To assemble the datasets, we first solve the TOPs from their nominal initial states, as described in Sections III-B and III-C. From these initial states, we employ Algorithm 2 to obtain a set of solutions $\boldsymbol{T}$ from various other initial states, under the nominal homotopy parameter configurations: $\alpha=0$ and $\alpha=\beta=0$ for $P_{0}$ and $P_{1}$, respectively. We then employ Algorithm 1 to obtain solutions across the homotopy parameters’ ranges. Finally, we numerically integrate444We use an adaptive-stepsize Runge-Kutta method of order 8(5,3) [30]. these systems with the solutions’ parameters to obtain datasets of the form $D=\{(\boldsymbol{s},\alpha,\boldsymbol{u})_{i}\forall i\}$ — $D_{0}$ with $945$ trajectories ($i=577,480$) from $P_{0}$, and $D_{1}$ with $5,525$ trajectories ($i=2,124,668$) from $P_{1}$ ($\beta=1$).

IV-B Neural network architectures

We augment the state-feedback controller formalism to synthesise multilayer-perceptron (MLP) policies of the form $\pi(s,\alpha)\mapsto\boldsymbol{u}^{\star}$. We denote $\pi$’s hidden architecture with $m\times n$, describing $n$ hidden layers, each having $m$ nodes. For every hidden layer we sequentially apply the standard NN operations: linear transformation, layer normalisation, and softplus — [25] indicated that softplus activations result in smoother and more meaningful control strategies. For brevity, we denote $P_{0}$’s and $P_{1}$’s instances of $\pi$ as $\pi_{0}$ and $\pi_{1}$, respectively. We apply the hyperbolic tangent in the last layer and use its output to approximate the controls of $P_{0}$ and $P_{1}$: $\pi_{0}(\boldsymbol{s},\alpha)\in[-1,1]=u$ and $\pi_{1}(\boldsymbol{s},\alpha)\in[-1,1]^{3}\mapsto[u~{}\phi~{}\theta]^{\intercal}\in[0,1]\times[0,\pi]\times[0,2\pi]\mapsto u\cdot[\sin(\theta)\cos(\phi),\sin(\theta)\sin(\phi),\cos(\theta)]^{\intercal}=u\cdot\hat{\boldsymbol{u}}$, respectively.

IV-C Evaluation

For $\pi_{0}$ and $\pi_{1}$, we consider the hidden architectures: $50\times 2$, $50\times 4$, $100\times 2$, $100\times 4$. We train these MLPs on their respective datasets — $D_{0}$ and $D_{1}$ — for $10,000$ episodes with the Adam optimiser [31], with a learning rate of $10^{-3}$, a decay rate of $10^{-5}$, and a mean-squared error (MSE) loss function. At each training epoch, we randomly sample $20,000$ datapoints, reserving $10\%$ for validation. The training history and regression performances are shown in Figure 3(c) and Table I, respectively.

Since success in the regression task does not necessarily mean that the policy-controlled systems will behave as expected, we also evaluate the policy trajectory optimality metric, described in [25], indicating the optimality-difference between a policy-controlled and optimal trajectory from an initial state. From the systems’ nominal initial states, we evaluate this metric across the hidden architectures and a range of homotopy parameters — shown in Table II.

Across all architectures, we find that $\pi_{0}$ and $\pi_{1}$ perform well in the regression task and result in trajectories that are near-optimal with respect to the true optimal, across a range of homotopy parameters. We observe that increasing node breadth and number of layers generally results in slightly better regression performance as well as trajectory optimality.

V Conclusions

We presented a novel approach to synthesising objective-conditioned near-optimal policies through trajectory optimisation, homotopy, and imitation learning. We further demonstrated that these state-feedback policies produce trajectories that are near-optimal, and can adapt their behaviours online to changing objectives, due to unexpected events or user-defined preferences.

Acknowledgement

This work was supported by Stiftelsen for StrategiskForskning (SSF) through the Swedish Maritime Robotics Centre (SMaRC) (IRC15-0046).