A Staged Approach to Evolving Real-world UAV Controllers

TL;DR

This paper presents a two-stage evolutionary approach for developing more general UAV controllers by initially evolving controllers with physical constraints and then refining them in a less restricted environment, improving generalization and efficiency.

Contribution

The paper introduces a novel two-stage evolution method that enhances controller generalization for real-world UAVs by combining constrained and less restricted optimization phases.

Findings

Two-stage approach yields more general controllers in fewer generations.

Controllers evolved with the method perform better in real-world hover tasks.

Sensitivity analysis confirms robustness of the evolved controllers.

Abstract

A testbed has recently been introduced that evolves controllers for arbitrary hover-capable UAVs, with evaluations occurring directly on the robot. To prepare the testbed for real-world deployment, we investigate the effects of state-space limitations brought about by physical tethering (which prevents damage to the UAV during stochastic tuning), on the generality of the evolved controllers. We identify generalisation issues in some controllers, and propose an improved method that comprises two stages: in the first stage, controllers are evolved as normal using standard tethers, but experiments are terminated when the population displays basic flight competency. Optimisation then continues on a much less restrictive tether, effectively free-flying, and is allowed to explore a larger state-space envelope. We compare the two methods on a hover task using a real UAV, and show that more general solutions are generated in fewer generations using the two-stage approach. A secondary experiment undertakes a sensitivity analysis of the evolved controllers.

Tables

Table 1. Table 1: Waypoints for OSE (left in cell) and TSE (right in cell). Position error is calculated from the centre of the UAV.

t (s)	N (m)	E (m)	h (m)	$\psi$
0	0, 0	0, 0	0.2, 0.2	40, 40
10	0.06, 0.15	-0.06, -0.15	0.2, 0.25	-5, -5
20	-0.06, -0.15	0.06, 0.15	0.2, 0.4	40, 40
30	0.06, 0.15	0.06, 0.15	0.2, 0.25	85, 85
40	0.06, 0.15	0.06, 0.15	0.2, 0.4	40, 40
50	0, 0	0, 0	0.2, 0.25	40, 40
60	stop	stop	stop	stop

Equations

$$K_{\mathrm{pV}}=K_{\mathrm{iV}}=K_{\mathrm{dV}}=(0,\frac{l_{\mathrm{cmd}}}{l_{\mathrm{er}}}]$$

$$o(t)=K_{p}e(t)+K_i{\int }_0^{t}e(\tau )d\tau +Kd\frac{d}{dt}e(t)$$

$$f_{cycle}=f_a+f_{vh}+f_{vv}+f_h+f_{\psi }+f_p+f_{\omega }+f_l$$

$$\mathbf{v}=r3+F(r1-r2)$$

$$\mu \leftarrow \mu \ast e^{N(0,1)}$$

$$\frac{T}{T_{\infty }}=\frac{1}{1-{\left(\frac{R}{4z}\right)}^2}$$

Full text

††journal: ArXiV submission

1 Introduction

Rapidly-advancing Additive Manufacturing technologies are leading a shift from mass production of identical simulacrums to provision of one-off, bespoke systems. Robotics is particularly suited to benefit from this shift; permitting rapid prototype-and-test of bespoke robotic systems with specialised morphologies to achieve heightened environmental/task-specific performance. In such an approach, robots may appear in a wide variety of different morphological compositions, with varied payloads. Behaviours must be generated to fully harness this diversity of morphologies. Evolutionary Algorithms (EAs) are ideal for this task as they are problem- and platform-agnostic optimisers that can account for morphology, payload, environmental and task requirements (Eiben and Smith, 2003).

Although EAs are a promising suite of optimisers for this task, their dependence on stochastic search processes leads to difficulties when optimising on real hardware, mainly around time requirements, repeatability, and accurate fitness assessment. As a consequence, the push towards bespoke robotics requires a concurrent push towards automated learning facilities (herein referred to as testbeds (Heijnen et al., 2017; Faina et al., 2017)), where robots will have their behaviours and morphologies optimised. Key characteristics of such testbeds include: closed loop operation (no human intervention required), robot-ambivalent optimisation, and the ability to produce robots that work in the real world.

UAVs are an ubiquitous example of general-purpose robots being required to perform a rapidly-expanding variety of increasingly specific tasks (e.g., tracking, mapping, inspection, repair, and delivery). As these tasks become more challenging (or more niche), the ability of a general-purpose UAV to perform well significantly decreases. As such, UAVs are highly likely to benefit from the development of systems that permit their specialisation for increased task performance, an increased range of deployment scenarios, and more favourable mission outcomes.

To address this need, we recently introduced a testbed (Fig. 1) that optimises controllers for arbitrary UAVs (Howard and Merz, 2015; Howard, 2017a)111Herein we use ’UAV’ to refer to any hover-capable multirotor, with an airframe size $<800mm$. To date, we have already demonstrated (i) repeatable evolutionary experimentation to test hypotheses in the real world, and (ii) optimizing UAVs with unconventional payloads and physical setups that would not be otherwise flyable.

This article covers the development of our testbed from a prototyping to a deployment stage, ensuring that our testbed reliably generates controllers that are more generalised to real-world conditions. The main scientific contributions of this work are:

The development of a two-staged technique (a form of incremental learning (Rossi and Eiben, 2014)) that works in hardware on real UAVs and reliably generates controllers suitable for real UAV deployments; 2. 2.

Statistical comparisons between the new and old methods, and; 3. 3.

A comprehensive analysis of the best controllers to ascertain how close they are to optimal values.

We pose the following research questions:

Given that evolutionary performance is critically dependent on the amount of state space available (Sutton and Barto, 1998), can we improve the optimisation process by increasing the amount of state space experienced during training enough that the evolved controllers can generalise to unseen waypoints, as found in real world missions? Can we balance this desire with the requirement for a safe testing platform. 2. 2.

Can we show that controllers evolved through this improved optimisation are optimal for the mission/payload/UAV considered?

Results will guide the development of further research testbeds, and, as all current learning techniques are sensitive to the state-space the algorithm experiences, are broadly applicable to robot learning in general.

1.1 Background

To place our work in the broader context of relevant literature, we now briefly summarise research in evolving UAV control, and robot testbed development.

1.1.1 Evolving UAV controllers

EAs typically require a substantial number of evaluations to discover promising solutions due to the underlying mechanisms of iterative, population-based, stochastic search. In addition, early generations are largely comprised of low-performance controllers. Simulation, e.g., (De Nardi et al., 2006; Howard and Elfes, 2014; Koppejan and Whiteson, 2009) is therefore the preferred methodology when evolving UAV controllers, as (i) it cannot physically damage the UAV, (ii) the environment is fully controllable, and (iii) evaluations are parallellisable and generally run faster than realtime.

All simulations, no matter how complex, necessarily abstract reality to some degree. As such, a continuing research focus is to cross the ’reality gap’ (Jakobi et al., 1995), and replicate simulated performance on real robot. Despite continuing efforts, including coevolutionary model learning (Holland and Nardi, 2008), specifically selecting for transferability (Koos et al., 2010), and manual post-evolution rule-tweaking (Scheper et al., 2016), the only way to guarantee performance in reality is to evolve in reality.

Real-world attempts to evolve UAV controllers are few in number due to inherent difficulties of stochastic optimisation in hardware. Control of a blimp is successfully evolved (Floreano et al., 2003) in a large, open space, but the slow dynamics of the blimp simplifies the control problem, as well as trivialising recovery from dangerous states. Height and yaw control of a miniature helicopter (Gongora et al., 2009) are evolved, although other degrees of freedom are neglected to prevent suboptimal controllers from potentially damaging the UAV. Although there have been some attempts to perform exhaustive analysis of evolved solutions on real systems (Gongora et al., 2009), it is not a well-studied area of research.

1.1.2 Robotic testbeds

Real-world evolution implies the need for a robotic testbed, as defined in in Section 1. Testbeds provide a controllable, measurable evaluation environment. They are chiefly used for field testing; ensuring that the system can handle real-world (i) sensor and actuator noise (How et al., 2008), (ii) software issues (Nishiwaki et al., 2000), (iii) environmental conditions (e.g., underground (Acar et al., 2001), space (Samuele et al., 2010)), and (iv) mission requirements (e.g., multi-robot coordination (Acar et al., 2001)). They focus on state estimation to evaluate performance, and the robots are typically manually controlled for missions with a duration from 5 minutes to a couple of hours.

Performing iterative optimisation in addition to evaluation imposes additional requirements on the testbed. Such testbeds should consider (i) repeatable evaluations to ascertain fair, reliable optimisation scores, (ii) automated experimental management, to reduce the human requirement through multiple optimisation generations and remove the requirement to manually operate the robot under evaluation, and (iii) provision of power, again to reduce human intervention whilst making the optimisation tractable in time.

One of the earliest examples (1994) approximated a differential-drive ground robot with a gantry-mounted camera, thus removing issues with motor wear and power supply (Harvey et al., 1994). Technology advancements later allowed real robots to evolve on a testbed — examples include gait optimisation for legged robots (Yosinski et al., 2011; Degrave et al., 2015). More recent work includes multi-objective optimisation of legged robot behaviours as a fully closed-loop system (Heijnen et al., 2017), and a testbed that can alter its layout via a robot arm and vision/marker system (Faina et al., 2017), to evolve controllers for difficult environments by incrementally bootstrapping them in simpler ones.

Aside from (Heijnen et al., 2017), human intervention is required to, e.g., change batteries, reset the robot if it moves out of bounds, etc. One notable example successfully evolves hardware UAV controllers (Ghiglino et al., 2015), however state estimation requires an expensive infra-red tracking system, and frequent human intervention is required to change batteries.

Our testbed is fully described in (Howard, 2017b, a), and is unique in that it can run back-to-back optimisations on a real UAV in a fair, repeatable manner, indefinitely, and without human intervention. High-performance controllers are generated for real UAVs that are specific to the mission, payload, and hardware state of the UAV being optimised. The most recent iteration of the testbed (Howard, 2017b) optimises a controller for an arbitrary UAV (tested on two different quadrotors and a hexacopter) from scratch in 3-4 hours. The testbed has also optimised control for a UAV carrying a heavy off-centre payload; something commercially-available self-tuning autopilots cannot do (Howard, 2017a).

As evaluations using the testbed occur in real-time, subsequent improvements focused on reducing the number of evaluations required. Self-adaptive mutation rates, e.g.,  (Rechenberg, 1973), are used as a form of experimental process optimisation, and adapt to the experimental specifics (UAV, payload, and environment) which significantly reduces the number of evaluations needed compared to algorithmically-determined static rate settings (Howard, 2017a). Rate restart strategies (Howard, 2017b) make experiment times more predictable by reducing variance in the total number of evaluations required.

This research focuses on ensuring the feasibility of the controllers in real-world settings, including (i) generalisation analysis of evolved controllers, and (ii) methods that improve the ability of the controllers to function in the real world. This work represents the final, critical, step before deployment for our UAV operations.

2 Material and Methods

2.1 Testbed

The testbed, which is exhaustively described in (Howard, 2017a), comprises a solid floor which is covered with foam matting. Before an experiment starts, the target UAV is set up in the testbed (Fig. 1). Flipping (tilt angles $>60^{o}$), and excessive rotation ($\pm 160^{o}$) of the UAV are prohibited by two nylon wires, which are attached to the UAV and the centre of the testbed’s floor. A camera, mounted centrally on a mesh-covered metal frame, provides position estimates. Wind disturbances of $\approx$5m/s are provided by a fan, with a total traversal angle of $120^{o}$ and oscillation period of 10 seconds. A 24V power tether permits continuous operation, and a serial cable connects to the host PC, which manages and monitors experiments using the real-time Extended State Machine (ESM) framework (Merz et al., 2006). Our testbed provides a comprehensive set of functions to enable closed-loop, automated UAV controller generation:

• •

Accurate state estimation for control and evaluation, using a conventional camera (position) and AHRS (attitude), with no expensive marker-based tracking required.

• •

Performance evaluation of controllers in reality, in a confined space.

• •

Robot-agnostic controller evolution for any robot capable of hover, with no models or simulator required, automatically taking into account payload and hardware variability.

• •

Continuous experimentation, including 24/7 evolution, experimental monitoring and statistics recording or repeatable, fair tests. The testbed can carry out back-to-back experiments for over a week without human intervention.

• •

Health monitoring of dangerous states, with UAV recovery. Error detection algorithms identify and and re-run erroneous trials222Typical causes include, e.g., the tracking LED being obscured, or data link errors.

• •

Environmental interactions, e.g., wind, can be included in the evaluations.

2.2 Controllers & State Estimation

The goal is to optimise the PID controller of the UAV, which has a nested structure shown in Fig. 2. PID control aims to minimise the error of the UAV during flight, which we take as the deviation between the UAVs actual pose and desired pose. Pose consists of 6 elements, or Degrees of Freedom (DoF); roll $\phi$, pitch $\theta$, yaw $\psi$, height $h$, and lateral position in North $p_{n}$ and East $p_{e}$. Three gains $K_{p}$, $K_{i}$, and $K_{d}$ control the error response per DoF. Each controller is therefore parameterised by a gain set of 18 real-valued gains (6 DoFs, 3 gains per DoF).

At the start of an experiment, we bootstrap our population by repeatedly randomly initialising gain sets within allowed ranges (1). Here $l_{\mathrm{cmd}}$ is a generalised maximum possible command (Pulse Width Modulation, or PWM) for each parameter $V$: $l_{\mathrm{cmd}}$=300 for $\phi$, $\theta$, $\psi$, and $h$, and 20cm for $p_{n}$/$p_{e}$. The gain set is accepted into the initial population if it allows the UAV to stay in the air for $0.2$s when attempting to hover at a height of 10cm with all over DoFs neutral. When the population size reaches $N=20$, the first generation begins.

[TABLE]

We evaluate each gain set by loading it into the PID structure. To ensure a fair, repeatable test, the UAV is reset to a designated start position (in the centre of the floor area with \psi$$\pm60o) between evaluations. Once reset, the UAV attempts to fly a predefined waypoint set, where each waypoint defines a desired setting of position and yaw: ($\psi_{\mathrm{sp}}$, $P_{\mathrm{nsp}}$, $P_{\mathrm{esp}}$, $h_{\mathrm{sp}}$). To successfully minimise error in the position waypoints, roll and pitch error must also be minimised; hence all 6 DoFs are optimised by these waypoints. Note that the fan is not reset between trials; this prevents overspecialisation but provides different conditions to the controllers. To ensure a fair test we re-evaluate any successful controller 3 times in total to prove generality to wind conditions, and use an average fitness score (see Section2.3).

As our controller responds to error, we must continually estimate the UAV’s state to compare to our desired waypoint. An onboard Inertial Measurement Unit (IMU, a Microstrain GX4-25) estimates roll $\phi$, pitch $\theta$, yaw $\psi$ (250Hz), and height $h$ (20Hz) when combined with an optical range finder. An external camera estimates position in North $p_{n}$ and East $p_{e}$ at 60Hz. Angular velocities $\dot{\phi}$, $\dot{\theta}$, $\dot{\psi}$ (250Hz) are derived from two consecutive Euler angles, and position velocities $\dot{p_{n}}$, $\dot{p_{e}}$, $\dot{h}$ (60Hz) calculated through a linear regression of five consecutive position estimates. Height is processed through a complimentary filter; the Kalman filters integrated into the IMU were bypassed and manually re-implemented to give us full control and observability of the system.

The PID uses the state estimate to calculate errors $e$ in all DoFs. Errors are taken as the difference between the waypoint value and the UAV’s current value per DoF. Each error is limited (10cm for $h$, 15o for attitude, 15cm for $p_{n}/p_{e}$) before being input to the PID.

The PID takes these error signals, and in response produces four outputs ($\delta_{\phi}$, $\delta_{\theta}$, $\delta_{\psi}$, and $\delta_{T}$), which represent commanded changes in roll, pitch, yaw, and thrust, scaled to fall in the range of possible motor PWMs ($l_{ul}$=1090 and $l_{uh}$=1950). Error response follows (2), where $o$ is the PID output, $t$ is the instantaneous time, and $\tau$ is the integration time step from 0 to $t$. These outputs are passed to a linear mixer, which provides one control command $u$ per motor $m$, $u_{1}$ to $u_{m}$ at 250Hz.

[TABLE]

2.3 Evolutionary Algorithm

During evaluation, a controller accumulates fitness (initially 0) at 250Hz by adding a per-Hz fitness measure $f_{cycle}$ (max. 10) to a running total $f$ (max. 150,000). $f_{cycle}$ is a compound fitness function (3), with components measuring the error in pitch/roll ($f_{a}$), horizontal and vertical velocity ($f_{vh}$, $f_{vv}$), height ($f_{h}$), yaw ($f_{\psi}$), horizontal position ($f_{p}$), pitch/roll rates ($f_{\omega}$), and staying within controller limits ($f_{l}$). Full calculations are shown in Appendix A.

[TABLE]

Once each gain set has a fitness, we use self-adaptive Differential Evolution (DE) (Storn and Price, 1997) to generate new gain sets. DE is a state of the art optimiser of real-valued vectors, such as our PID gain sets (Chiha et al., 2012; Biswas et al., 2009), and has seen previous success in a robotic context (Moravec and Pošík, 2014); further justification for using DE can be found in (Howard, 2017a).

A donor vector v is created for each gain set in the population p following (4). $F$ is a differential weight, and r1, r2, and r3 are the gain sets of three unique randomly-selected individuals.

[TABLE]

A child controller c is created for each parent by probabilistically replacing elements of p with those of v. For each vector index $i$, $c_{i}$ = $v_{i}$ if $i==R$ or $rand$ $<$ CR, otherwise $c_{i}$ = $p_{i}$. $rand$ is a uniform-random number in range [0,1], and CR is the crossover rate. $R$ is a vector index, selected randomly per c, ensuring c contains at least one element of v. Each child is evaluated and assigned a fitness $f$, and replaces its parent if it is fitter. This concludes a generation. Each subsequent generation involves creation of one child per parent, evaluating and assigning fitness, and creating the next generation by selecting parents and children based on fitness, as above.

Self-adaptation based on an Evolution Strategy (Rechenberg, 1973) allows the testbed to tailor the learning process to the UAV, mission, and payload under consideration (Howard, 2017a). New population members random-uniformly initialise their CR and F, respecting bounds for CR=[0,1], and F=[0,2]. Each ${c}$ copies CR and F from its ${p}$, and modifies them following (5), before using them to alter its controller.

[TABLE]

To prevent the learning process being stuck due to suboptimal rate settings, the rates of a given parent are uniform-randomly reinitialised if it does not create a fitter child within 5 generations333Selected to balance search stability and convergence times following a parameter sweep.. This combination of self-adaptation with restarts has been shown to promote high-fitness controllers, while significantly reducing the number of generations required, when compared to optimally-set static rates (Howard, 2017b).

As a stochastic optimser, we must be mindful of testing gain sets that are harmful to the UAV. ESM monitors the UAVs behaviour throughout an evaluation, and safely terminates as required to preserve the UAV. Terminated controllers are assigned their currently-accumulated fitness. Termination occurs when;

• •

angles $\phi>60^{o}$, $\theta/\psi>15^{o}$,

• •

any angular rate$>250^{o}/sec$,

• •

horizontal velocities $v_{n}$/$v_{e}>$ 50cm/s,

• •

vertical velocity $v_{h}>$25cm/s,

• •

total current draw $>20A$,

• •

if the UAV does not take off within 5s of evaluation start,

• •

or if the UAV pulls on the tether ($h>$18cm). 444This latter criterion prevents the UAV from cheating by using the tether to ’balance’ itself.

Any controller that successfully completes the entire waypoint set once is re-evaluated twice more, and assigned the average fitness of all three runs. If the controller completes all three repeats, it is called a success. The experiment ends when we have an entire population of successful controllers.

2.4 Experimental

Our testbed is designed to optimise UAV controllers for real-world flight. Design decisions to date support this goal: optimising real UAVs, with real payloads, in strong, realistic wind disturbances. However, stochastic optimisation of hardware UAV controllers for real-world conditions is challenging, as the amount of state space exploration must be balanced with the requirement for non-destructive controller evaluation. Post-hoc analysis of some previously-generated controllers showed degraded performance when the UAV was removed from its tether and allowed to free-fly inside the test bed. There are two main reasons for this. Firstly, the state space available to the UAV may be too limiting, resulting in a lack of generalisation. Secondly, controllers optimised in the ’ground’ effect555when flying close the the floor, the ground deflects a propellers airflow, causing increased thrust nearer the ground for the same power input (Johnson, 2012), may perform sub-optimally when outside of it. A widely used ground-effect equation is shown in (6), where $R$ is the radius of the rotor, $z$ is the vertical distance to the ground, $T$ is the thrust in ground effect, and $T_{\infty}$ is the thrust produced at the same power outside of the ground effect.

[TABLE]

It follows that ground effect is negligible ($T\approx T_{\infty}$) when $z/R>$2666$>$4 in certain circumstances (Powers et al., 2013), and that the influence of the ground effect diminishes rapidly the UAV gets further from the ground (6). The simplest way to mitigate both of these factors is to increase the amount of flying area available to the UAV, so TSE appears to be a simple and viable route towards real-world flyability.

In our first experiment, we compare the original approach of using a single physical tether throughout the experiment (One-Stage Evolution, OSE) with a new incremental method that partially evolves controllers with the original tether, before switching to a more permissive tether to complete the evolution in a larger state space (Two-Stage Evolution, TSE), as shown in Fig. 3.

Controllers are optimised on a wind-affected hover scenario, with a total evaluation length of 60s. and waypoint transitions every 10s. Transitions are rate-limited: $\psi=30^{o}/s$, N/E=0.1m/s, h=0.2m/s. The waypoints are selected to excite all of the UAV’s six DoFs by requiring controlled movements in each, and shown in Table 1.

The OSE tether is set to 22cm, the maximum length that prevents the UAV from flipping. The TSE tether is set to 62cm, the maximum length that prevents the UAV from contacting the fan, to more closely represent free flight. The experiments proceed as follows:

• •

OSE controllers fly the OSE waypoint set until all controllers are successful.

• •

TSE controllers initially fly the OSE waypoint set until the population contains its first successful controller. A new population ($N$=20) is created from copies of this controller, with each gain subject to uniform noise of $\pm$25% of the total range of that gain when copied. The UAV is transferred to a longer tether (Fig. 3), and experimentation continues on TSE waypoints until all controllers are successful.

Ten experimental repeats of OSE and TSE are run, with statistical significance assessed with a Mann-Whitney U-test (which does not require normally-distributed samples). OSE and TSE repeats both have a population size of 20 controllers, and are run until the entire population is filled with successful controllers.

3 Results and Discussion

As we operate in hardware and are limited to real-time evaluations, reducing the number of evaluations required to optimise our controllers is particularly interesting to us (see, e.g., (Howard, 2017b)). Here, we see that TSE lowers the mean convergence generation from 45.6 (OSE) to 32.5 (Fig. 4, which is statistically significant (p$<$0.05) and indicates that TSE is a viable technique for reducing the number of generations required to perform an experiment. Although generally more closely distributed around the mean, standard deviation for TSE convergence is higher than OSE (12.32 vs. 10.13) due to a single outlier at generation 64. Visualising individual experimental fitness progressions reinforces this. Although TSE typically converges faster than OSE (Fig. 5(a) vs Fig. 5(b)), the outlier in Fig. 5(b) (a grey line) lags far behind the other repeasts. Despite this, TSE is clearly preferable in terms of convergence.

Both OSE and TSE produce high fitness controllers (Fig. 5(a) and (b)). Final fitness values are quantatively similar, but cannot be directly compared as consecutive TSE waypoints are typically further from each other than OSE waypoints. Mean fitness for OSE and TSE controllers was 99234 vs. 97184, 93090 vs. 90880, and 82786 vs. 82735, for best, average, and worst fitness respectively.

As a more representative test of the generalisation ability of the controllers, we run the highest fitness controller per repeat on an unseen waypoint set. Each controller in this set would be selected for any real-world deployment, based on their performance in the first experiment.

This waypoint set has the following transitions, with one transition per 10s as before. The four array elements respectively represent position in N(m) and E(m), height h(m), and yaw $\psi$: ( [0.0, 0.0, 0.4, -10], [-0.25, 0.25, 0.2, 45], [0.25, -0.25, 0.4, 85], [0.25, 0.25, 0.4, 85], [0.0, 0.0, 0.3, -10], [-0.25, -0.25, 0.4, 45]). The UAVs are placed on the longer stage 2 TSE tether. Each controller is evaluated 20 times to generate reliable results. The UAV flies away from significant ground effect.

Results are shown in Fig. 6. Notably, half of the OSE controllers (those from repeats 2, 3, 5, 8, and 10, Fig. 6(a)) are seen to struggle when flying the TSE waypoints on a longer tether. This suggests that the OSE setup does not properly expose the required state space to the controller, and as such some OSE controllers struggle when flying the new waypoint set. All TSE controllers (Fig. 6(b)) can handle the TSE waypoint set, displaying a consistently higher fitness than OSE controllers.

It was observed that many OSE controllers from the aforementioned repeats oscillate, which causes premature fight termination and the resulting low fitness values. This is likely due to the restricted state space available during OSE experiments, and indicates that OSE is not the best way to evolve controllers for real-world scenarios. TSE is shown to be more successful in producing such controllers by being able to provide a more expansive state space during evolution. However, it should be noted that both OSE and TSE are capable of generating real-world flying controllers (OSE repeats 1, 4, 6, 7, and 9 in Fig. 6(a)).

The best fitness score overall for OSE was 102363.1, and for TSE 100188.1. Averaged across the entire experimentation on the new waypoint set, OSE fitness was 69810.6, and TSE fitness was 91839.7. Averaging each controller’s repeats individually, this is statistically significant.

Next, we assess the affect of the TSE experimental procedure on our self-adaptive mutation strategy. The mean crossover rate $CR$ varies between 0.482 and 0.553 for the first stage of TSE, raising to a maximum of 0.589 for the second stage (Fig. 7(a)). Context-sensitive adaptation of the learning rates is seen between stage 1 and 2 in TSE, in particular the rise in $CR$ is a response to the reseeding of controller parameters, which subsequently requires more parent-child variance. Self-adaptive rates also vary between OSE and TSE, showing again that rates can adapt to experimental setup as required.

$F$ has more effect on the evolutionary process than $CR$; as such the setting of $F$ between stage 1 and 2 in TSE is more pronounced than the corresponding change observed between stages for $CR$, which agrees with the literature of rate setting in DE(Storn and Price, 1997), and supports previous results for our testbed (Howard, 2017a). $F$ self-adapts from 0.57 to 0.672 within 2 generations of stage 2 commencing, which induces more variance in the search process, either (i) in response to the increased search space, or (ii) to combat the increased parameter convergence in the stage 2 population. This feature is key to TSE achieving more expedient convergence than OSE. Stage 2 $F$ rates are statistically different from both stage 1 and OSE $F$ rates.

3.1 Controller Optimality and Sensitivity

Next, we incrementally step through a range of feasible gain settings to show the optimality of our discovered controllers. We focus on height control as it is most critical to the UAVs ability to accumulate fitness. Mapping the effects of changing gains on controller fitness allows us to (i) confirm that our testbed produces the highest fitness controllers possible, and (ii) estimate how brittle those controllers are. Note that without the testbed, this mapping would be extremely taxing to achieve.

The highest fitness controller from TSE is used; we take the three height gains for that controller (P, I, and D), and for all perturbations, hold one gain static whilst systematically varying the other two gains within the following ranges:

• •

P gains step from -0.1 to -1, in increments of 0.1.

• •

I gains step from -0.15 to -1.5, in increments of 0.15.

• •

D gains step from -0.05 to -0.5, in increments of 0.05.

This gives us ten possible settings per gain. Ranges were chosen to fall centrally around the corresponding value from the best evolved controller, whilst covering a wide range of possible rate settings. Each gain combination is run 20 times on the final waypoint set using the TSE tether, and fitness values are recorded and averaged. The fitness landscape produced by this exhaustive gain search (Fig. 8) shows performance degradation when gains diverge from their optimal values.

Notably, the maximum fitness values discovered through this search reach within 3$\%$ of the best evolved controller’s fitness; in other words, evolution consistently finds the best controller settings. Owing to the repeatability offered by the testbed, we can confirm that our best evolved height controller lies within the discretised global maximum discovered through exhaustive search (although in significantly less time, with fewer evaluations required).

Mapping the search space in this way also allows us to easily see infeasible solution subspaces, e.g. low I (-0.15) where the UAV is unable to take off before being ‘timed out’ after 5s of inactivity, or low D (-0.05) gains causing trial-ending overshoots; which is an important feedback for future experimental design in terms of being able to ignore certain solution subspaces a priori. The complexity of the problem (through e.g., sensory state estimation, noisy fitness evaluations, and other real-world artefacts) is evidenced through the appearance of multiple disconnected local maxima, shown in Fig. 8(b) and (c).

3.2 Conclusions

In this paper we investigated a method to increase the real-world flyability of our UAV controllers by increasing the amount of state space, and reducing the ground effect. Wße compared our standard method of restrictive tethering (OSE) to a new method of partially evolving the controllers, and subsequently transferring to a longer tether (TSE) for extra exploration.

Statistical comparisons between OSE and TSE shows that TSE produces more general controllers than OSE. As such, TSE will be the new de facto technique as we continue our experimentation on the testbed. We note that this technique can potentially be extended with various tether lengths, where the incremental learning we demonstrate here allows for the safe transition to progressively larger and larger state spaces.

Finally, we used the testbed to perform an exhaustive gain analysis, and showed that our testbed successfully finds the global optimum gain settings for the scenario we investigated.

These findings motivate further development towards our final goal quickly optimising high-performance controllers for arbitrarily-configured UAVs, that are optimised to mission, payload, and morphology, and are guaranteed to work in the real world. In particular, we see an opportunity for our testbed in optimising the control and behaviour UAVs with arbitrary morphologies. Such UAVs could be designed through evolution, and come in a range of unconventional physical configurations that may not be easily modellable. Optimisation in our testbed guarantees real-world performance and does not require a model, and as such will be a prominent technique for allowing such UAVs to reach their potential.

4 Funding Sources

This research received funding from the CSIRO Office of the Chief Executive for the Postdoctoral position in Evolutionary Aerial Robotics.

5 Conflicts of Interest

Gerard David Howard declares no conflict of interest. Alberto Elfes declares no conflict of interest.

Appendix A: Fitness Function

\displaystyle f_{\mathrm{cycle}}=f_{\mathrm{p}}+f_{\mathrm{h}}+f_{\psi}+f_{\mathrm{a}}+f_{\mathrm{vh}}+f_{\mathrm{vv}}+f_{\mathrm{\omega}}+f_{\mathrm{l}}\\ f_{\mathrm{p}}=f\left(\sqrt{(p_{\mathrm{nsp}}-p_{\mathrm{n}})^{2}+(p_{\mathrm{esp}}-p_{\mathrm{e}})^{2}},l_{\mathrm{p}},l_{\mathrm{pc}}\right)\\ f_{\mathrm{h}}=f(|h_{\mathrm{sp}}-h|,l_{\mathrm{h}},l_{\mathrm{hc}})\\ f_{\mathrm{\psi}}=f\left(|\mathrm{wrap}(\psi_{\mathrm{sp}}-\psi)|,l_{\psi},l_{\psi\mathrm{c}}\right)\\ f_{\mathrm{a}}=f(|\phi_{\mathrm{sp}}-\phi|,l_{\mathrm{a}},l_{\mathrm{ac}})+f(|\theta_{\mathrm{sp}}-\theta|,l_{\mathrm{a}},l_{\mathrm{ac}})\\ f_{\mathrm{vh}}=f\left(\sqrt{v_{\mathrm{n}}^{2}+v_{\mathrm{e}}^{2}},l_{\mathrm{vh}},l_{\mathrm{vhc}}\right)\\ f_{\mathrm{vv}}=f(|v_{\mathrm{v}}|,l_{\mathrm{vv}},l_{\mathrm{vvc}})\\ f_{\mathrm{\omega}}=f(|p|,l_{\omega},l_{\omega\mathrm{c}})+f(|q|,l_{\omega},l_{\omega\mathrm{c}})\\ f_{\mathrm{l}}=1-\frac{\mbox{number of controllers exceeding l_{\delta} limits}}{4}\\ f(e,l,l_{\mathrm{c}})=\begin{cases}\mathrm{max}\{\frac{l-e}{4(l-l_{\mathrm{c}})},0\},&\mbox{if }e>l_{\mathrm{c}}\\ \frac{3(l_{\mathrm{c}}-e)}{4l_{\mathrm{c}}}+\frac{1}{4},&\mbox{otherwise}\end{cases}\\ \mathrm{wrap}(\gamma)=\mathrm{atan2}(\sin(\gamma),\cos(\gamma))

Symbol Definitions - Fitness Function

$\displaystyle f_{\mathrm{cycle}}:\mbox{fitness for one control cycle}\\ f_{\mathrm{p}}/f_{\mathrm{h}}:\mbox{fitness for position/height tracking}\\ f_{\psi}:\mbox{fitness for yaw angle tracking}\\ f_{\mathrm{a}}:\mbox{fitness for pitch and roll angle tracking}\\ f_{\mathrm{vh}}/f_{\mathrm{vv}}:\mbox{fitness for low horizontal/vertical velocity}\\ f_{\mathrm{\omega}}:\mbox{fitness for low pitch and roll rates}\\ f_{\mathrm{l}}:\mbox{fitness for staying within controller limits}\\ p_{\mathrm{n}}/p_{\mathrm{e}}:\mbox{north/east position}\\ p_{\mathrm{nsp}}/p_{\mathrm{esp}}:\mbox{north/east position setpoint}\\ h:\mbox{height above ground}\\ h_{\mathrm{sp}}:\mbox{height setpoint}\\ \phi/\theta/\psi:\mbox{roll/pitch/yaw angle}\\ \phi_{\mathrm{sp}}/\theta_{\mathrm{sp}}/\psi_{\mathrm{sp}}:\mbox{roll/pitch/yaw setpoint}\\ v_{\mathrm{n}}/v_{\mathrm{e}}/v_{\mathrm{v}}:\mbox{north/east/vertical velocity}\\ p/q/r:\mbox{roll/pitch/yaw rate}$

\displaystyle l_{\mathrm{p}}:\mbox{position error range (0.2m)}\\ l_{\mathrm{pc}}:\mbox{core position error range (0.07m)}\\ l_{\mathrm{h}}:\mbox{height error range (0.2m)}\\ l_{\mathrm{hc}}:\mbox{core height error range (0.03m)}\\ l_{\psi}:\mbox{yaw error range (30{}^{o})}\\ l_{\psi c}:\mbox{core yaw error range (10{}^{o})}\\ l_{\mathrm{a}}:\mbox{attitude angle error range (15{}^{o})}\\ l_{\mathrm{ac}}:\mbox{core attitude angle error range (3{}^{o})}\\ l_{\mathrm{vh}}:\mbox{horizontal velocity error range (1 m/s)}\\ l_{\mathrm{vhc}}:\mbox{core horizontal velocity error range (0.2m/s)}\\ l_{\mathrm{vv}}:\mbox{vertical velocity error range (1 m/s)}\\ l_{\mathrm{vvc}}:\mbox{core vertical velocity error range (0.1 m/s)}\\ l_{\mathrm{\omega}}:\mbox{pitch and roll rate error range (250{}^{o}/s)}\\ l_{\mathrm{\omega c}}:\mbox{core pitch and roll rate error range (30{}^{o}/s)}\\