Long-Duration Fully Autonomous Operation of Rotorcraft Unmanned Aerial Systems for Remote-Sensing Data Acquisition

TL;DR

This paper presents a fully autonomous rotorcraft UAS capable of long-duration outdoor operations for remote sensing, including autonomous mission execution and vision-based precision landing for energy replenishment, demonstrated over multiple hours and flights.

Contribution

It introduces a novel fully autonomous rotorcraft system with vision-based precision landing, enabling long-term outdoor operations without human intervention, a first in the field.

Findings

Successfully demonstrated 11-hour indoor flight sessions.

Achieved 4-hour outdoor flight with 22 flights and autonomous charging.

Validated high accuracy of vision-based landing across various altitudes.

Abstract

Recent applications of unmanned aerial systems (UAS) to precision agriculture have shown increased ease and efficiency in data collection at precise remote locations. However, further enhancement of the field requires operation over long periods of time, e.g. days or weeks. This has so far been impractical due to the limited flight times of such platforms and the requirement of humans in the loop for operation. To overcome these limitations, we propose a fully autonomous rotorcraft UAS that is capable of performing repeated flights for long-term observation missions without any human intervention. We address two key technologies that are critical for such a system: full platform autonomy to enable mission execution independently from human operators and the ability of vision-based precision landing on a recharging station for automated energy replenishment. High-level autonomous decision making is implemented as a hierarchy of master and slave state machines. Vision-based precision landing is enabled by estimating the landing pad's pose using a bundle of AprilTag fiducials configured for detection from a wide range of altitudes. We provide an extensive evaluation of the landing pad pose estimation accuracy as a function of the bundle's geometry. The functionality of the complete system is demonstrated through two indoor experiments with a duration of 11 and 10.6 hours, and one outdoor experiment with a duration of 4 hours. The UAS executed 16, 48 and 22 flights respectively during these experiments. In the outdoor experiment, the ratio between flying to collect data and charging was 1 to 10, which is similar to past work in this domain. All flights were fully autonomous with no human in the loop. To our best knowledge this is the first research publication about the long-term outdoor operation of a quadrotor system with no human interaction.

Equations

$$T_{{\mathrm{lt}}_i}=T_{\mathrm{cm}}^{-1}{T}_{{\mathrm{ct}}_i}.$$

$${\mathbf{q}}_{\text{align}}$$

$${\mathbf{p}}_{\text{align}}$$

$${\mathbf{x}}_{\text{state}}=(\underset{\text{Core state}}{\underbrace{{\mathbf{p}}_{\text{w}}^{\text{b}},{\mathbf{v}}_{\text{w}}^{\text{b}},{\mathbf{q}}_{\text{w}}^{\text{b}},{\mathbf{b}}_{\omega },{\mathbf{b}}_{\text{a}}}},{\mathbf{p}}_{\text{b}}^{\text{g}},{\mathbf{q}}_{\text{b}}^{\text{m}},{\mathbf{m}}_{\text{w}},b_{\text{p}},{\mathbf{p}}_{\text{b}}^{\text{p}}),$$

$$\mathrm{touchdown}=({\tilde{p}}_{\text{w,z}}^{\text{b}}-{\tilde{p}}_{\text{w,z}}^{\text{l}})<0.3\wedge |{\tilde{v}}_{\text{w,z}}^{\text{b}}|<\mathrm{0.1.}$$

$$\mathrm{touchdown}=\Vert {\tilde{\mathbf{v}}}_{\text{w}}^{\text{b}}{\Vert }_2<\mathrm{0.1.}$$

Full text

1 Introduction

1.1 Motivation

The Food and Agriculture Organization of the United Nations predicts that food production will have to increase by 70 percent by the year 2050 in order to feed a projected additional 2.2 billion people [Food and Agriculture Organization of the United Nations, 2009]. Precision agriculture, also known as smart farming, is a data-driven approach that helps to address this challenge by using sensing technology to increase farming efficiency. Data is collected for variables like plant and animal health, crop yields, organic matter content, moisture, nitrogen and pH levels. To complement sensors mounted on farming vehicles, fixed-wing and rotorcraft Unmanned Aerial Systems (UAS) can be equipped with RGB, thermal and hyperspectral cameras and flown over fields in order to create cost-effective and on-demand orthomosaic maps of biophysical parameters. The market for such robots is projected to be the second-largest in the commercial drone sector [Goldman Sachs, 2017].

Rotorcraft UAS are ideal assets for deploying sensors and instruments in a 3D environment. Because they can hover and fly slowly, rotorcraft enable pin-point data gathering and can be operated in dense environments such as under tree canopies that are inaccessible to traditional fixed-wing platforms. In the agricultural domain, an important application is the repeated acquisition of the same target to capture changes in environmental properties such as plant water usage over the diurnal cycle. This is of increasing importance for ecosystem monitoring applications where accurate measurements of plant health parameters are desired down to leaf-level resolution of individual plants.

To validate the feasibility of UAS remote sensing of relevant data at high spatial resolutions, extensive data acquisition trials were carried out in collaboration with NASA’s Jet Propulsion Laboratory environmental scientists. Figures 1a and 1b show a hexarotor with a six-band visible and near-infrared (VNIR) camera and a thermal infrared camera payload that was used during several campaigns at the West Side Research and Extension Center (WSREC) at Five Points, California, to capture the diurnal cycle of plant water usage. Figure 1c illustrates a stitched map of the normalized difference vegetation index (NDVI) as an example data product of one flight. The map’s pixel resolution of approximately 5 cm is beyond what can be currently achieved with orbital or traditional airborne instruments. The autonomy engine developed in this paper can be deployed on such a quadrotor as a potential future application.

1.2 Contributions

Because the typical flight endurance of a quadrotor system with payload is relatively short (typically 10-20 min), long-term operation currently requires humans in the loop for energy replenishment (e.g. battery charging). This makes continuous observations over long periods of time impractical. To solve this issue, we propose a system that is capable of fully autonomous repeated mission execution, including automated recharging to replenish its batteries after a data collection flight is completed.

Our main contribution is the development and outdoor deployment of a high-level autonomy engine that is executed on-board the UAS and which implements the capability of robust fully autonomous repeated data acquisition. This includes flight control for take-off and landing, GPS-inertial state estimation, system health monitoring including the execution of emergency behaviors, recharging and mission data handling. Once the autonomy engine is started, the system has the logic and software-hardware interfaces necessary to operate indefinitely without humans in the loop.

The system is novel by being human-independent and operating outdoors. To the best of our knowledge, there have been no publications to date on quadrotor-type UAS platforms capable of autonomous recharging which have been flown outdoors for extended periods of time. Hence, prior work that demonstrates UAS recharging has relied on the idealistic state estimation accuracy offered by indoor motion capture systems. Since GPS-only localization is not sufficiently accurate for precision landing on a charging pad, our work augments the sensor suite with a monocular camera. In this context, we present an extensive analysis seeking to find a landing pad fiducial marker layout that maximizes landing accuracy.

1.3 Notation

The following notation is used throughout the report. Scalars are lowercase (e.g. $a$), vectors are lowercase bold (e.g. $\bm{a}$) and matrices are uppercase (e.g. $A$). The position of frame $\{\text{b}\}$ in frame $\{\text{a}\}$ is written $\bm{p}_{\text{a}}^{\text{b}}$. The quaternion active rotation from $\{\text{a}\}$ to $\{\text{b}\}$ is written $\bm{q}_{\text{a}}^{\text{b}}$ and the corresponding rotation matrix is $C_{\bm{q}_{\text{a}}^{\text{b}}}$. The homogeneous transformation matrix $T_{\text{ab}}$ takes homogeneous vectors in $\{\text{b}\}$ to $\{\text{a}\}$. Raw measurement quantities carry a hat, e.g. $\hat{\bm{p}}_{\text{a}}^{\text{b}}$, while filtered quantities carry a tilde, e.g. $\tilde{\bm{p}}_{\text{a}}^{\text{b}}$.

1.4 Outline

This paper is organized as follows. Section 2 surveys existing work on long-duration autonomy and vision-based landing, the two key enabling technologies for our system. The hardware and software architectures are then described in Section 3. Sections 4, 5 and 6 detail the navigation, guidance and control subsystems respectively which enable the UAS to execute flight tasks. The autonomy engine itself is presented in Section 7. Field tests with the system in environments of increasing complexity, up to real world conditions, are then given in Section 8, followed by a discussion of future work and conclusion in Sections 9 and 10.

2 Related Work

In this section we review related work in our two main areas of contribution, the ability to replenish energy for long-duration autonomy and vision-based precision landing.

2.1 Long-duration Autonomy

Several rotorcraft UAS that are capable of autonomous, long-duration mission execution in benign indoor (VICON) environments have previously appeared in literature. Focusing on the recharging solution to extend individual platform flight time and a multi-agent scheme for constant operation, impressive operation times have been demonstrated ([Valenti et al., 2007]: 24 h single vehicle experiment; [Mulgaonkar and Kumar, 2014]: 9.5 h with multiple vehicles). Recharging methods vary from wireless charging [Aldhaher et al., 2017] to contact-based charging pads [Mulgaonkar and Kumar, 2014] to battery swap systems [Toksoz et al., 2011, Suzuki et al., 2011]. While wireless charging offers the most flexibility since no physical contact has to be made, charging currents are low resulting in excessively long charging times and are hence not an option for quick redeployment. However, interesting results have been shown in [Aldhaher et al., 2017] demonstrating wireless powering of a 35 g/10 W micro drone in hover flight. On the other end of the spectrum, battery swap systems offer immediate redeployment of a UAS, but require sophisticated mechanisms to be able to hot swap a battery, and a pool of charged batteries that are readily available. This makes such systems less attractive for maintenance free and cost effective long-term operation.

2.2 Vision-based Landing

Our work uses a downward-facing monocular camera to estimate the pose of the landing pad in the world frame using AprilTag visual fiducial markers [Wang and Olson, 2016]. We believe that a monocular camera is the cheapest, most lightweight and power efficient sensor choice. Alternatively, GPS and RTK-GPS systems suffer from precision degradation and signal loss in occluded urban or canyon environments [Jin et al., 2016]. Laser range finder systems are heavy and consume considerable amounts of energy. Last but not least, stereo camera setups have limited range as a function of their baseline versus vehicle-to-pad distance.

Several landing approaches exist in literature for labeled and unlabeled landing sites. [Yang et al., 2012] present a monocular visual landing method based on estimating the 6 DOF pose of a circled H marker. The same authors extend this work to enable autonomous landing site search by using a scale-corrected PTAM algorithm [Klein and Murray, 2007] and relax the landing site structure to an arbitrary but feature-rich image that is matched using the ORB algorithm [Rublee et al., 2011]. [Forster et al., 2014, Forster et al., 2015] use SVO to estimate motion from a downfacing monocular camera to build a probabilistic 2 dimensional elevation map of unstructured terrain and to detect safe landing spots based on a score function. [Brockers et al., 2011, Brockers et al., 2012] develop a fully self-contained visual landing navigation pipeline using a single camera. With application to landing in urban environments, the planar character of a rooftop is leveraged to perform landing site detection via a planar homography decomposition using RANSAC to distinguish the ground and elevated landing surface planes.

However, landing algorithms which build elevation maps and perform generic landing spot detection trade their robustness for the ability to land in unlabeled environments. By placing visual fiducial markers at or near the landing zone, one can use visual fiducial detector algorithms to estimate the landing pad pose more reliably, i.e. with less false positives and negatives.

Currently one of the most popular visual fiducial detectors and patterns is the AprilTag algorithm [Olson, 2011] which is renowned for its speed, robustness and extremely low false positive detection rates. The algorithm was updated by [Wang and Olson, 2016] to further improve computational efficiency and to enable the detection of smaller tags. AprilTag has been applied multiple times for MAV landing [Brommer et al., 2018, Ling, 2014, Kyristsis et al., 2016, Borowczyk et al., 2017, Chaves et al., 2015]. Similar visual fiducials are seen around the landing zones of both Amazon and Google delivery drones [Amazon, 2016, Vincent, 2017] as well as of some surveying drones [Wingtra, 2017]. Our approach is the same as that of [Brommer et al., 2018]. We improve over [Ling, 2014, Kyristsis et al., 2016] by using a bundle of several tags for improved landing pad pose measurement accuracy. [Borowczyk et al., 2017, Chaves et al., 2015] appear to also use a tag bundle, however they deal with a moving landing platform and feed individual tag pose detections into a Kalman filter. Our approach is to use a perspective-$n$-point solution to obtain a single pose measurement using all tags at once.

After obtaining raw tag pose estimates from the AprilTag detector, our approach uses a recursive least squares (RLS) filter to obtain a common tag bundle pose estimate. A relevant previous work on this topic is [Nissler et al., 2016] in which a particle filter is applied to raw tag detections and RLS is compared to RANSAC for bundle pose estimation. Because the demonstrated accuracy improvement is marginal (about 1 cm in mean position and negligible in mean attitude) and RANSAC has disadvantages like ad hoc threshold settings and a non-deterministic runtime, we prefer RLS for its simplicity. Nevertheless, RANSAC and its more advanced variations [Hast et al., 2013] can be substituted into our implementation. Other work investigated fusing tag measurements in RGB space with a depth component [Jin et al., 2017] with impressive gains in accuracy. One can imagine this approach benefiting landing accuracy at low altitudes, however downward-facing stereo camera mounting on drones raises several concerns like weight, vibration effects and space availability.

Our approach, however, is not limited to AprilTags, which can be substituted or combined with other markers for specific applications. A vast number of markers is available targeting different use-cases [Jin et al., 2016, Yang et al., 2012, Fiala, 2005, Olson, 2011]. Patterns include letters, circles, concentric circles and/or polygons, letters, 2 dimensional barcodes and special patterns based on topological [Bencina et al., 2005], detection range maximizing [Xu and Dudek, 2011] and blurring/occlusion robustness considerations [Bergamasco et al., 2016].

3 System Overview

Our system consists of two major components: the aerial vehicle with its on-board autonomy software (autonomy engine) and a landing station which integrates a charging pad, a base station computer for downloading data, and a wireless router for communication with the UAS while landed. To deploy the system, a user connects the landing station to a power outlet and places the UAS on top of the landing surface as shown in Figures 2 and 3. After providing a waypoint based mission profile, a start command commences the operation of the system. All actions of the UAS from then on are fully autonomous with no human involvement. The autonomy engine implements various mission specific behaviors to guide the vehicle, including health monitoring and emergency response that are explained in Section 7.

3.1 Hardware Architecture

Figure 4 illustrates the main hardware elements of our test UAS. Figure 5 displays the quadrotor on-board hardware architecture. Solid lines represent information flow and are labelled whenever the communication protocol is known, and dashed lines represent energy flow. The gray blocks in Figure 5 label the components that host the flight software developed in this work.

To minimize recharging time, we use a contact-based, commercially available charging solution [Skysense, 2018] to provide ample charging current to the on-board charging electronics. The charging pad consists of an array of individual contact patches that cover a $90\times 90$ cm flat area. Charging is triggered by the charging pad electronics once contact to the charger on-board the UAS is made via leg contacts. Charging status is monitored online by the autonomy engine, which prepares for the next flight once the battery is sufficiently charged. While landed, the vehicle connects to the base station computer via WiFi to downlink mission data and to receive eventually updated mission plans (e.g. mission-end command from a user).

3.2 Software Architecture

Figure 6 illustrates the full autonomy system as an information flow diagram. Sensors feed information to the Single Sensor Fusion (SSF) state estimator [Weiss et al., 2012], modified to handle multiple sensors, and to the visual landing navigation subsystems. These supply the control, guidance and autonomy algorithms with navigational information necessary for stabilization, trajectory generation and state machine transitions. The control subsystem is then responsible for trajectory tracking and desired motor speed computation.

3.3 Mission Architecture

The UAS executes a mission profile depicted in Figure 2. Before each take-off, the system initializes the on-board state estimator and passes a series of pre-flight checks that include testing for an adequate battery voltage level and motor nominal performance. The vehicle then performs a take-off maneuver by issuing a vertical velocity command to its low-level velocity controller to climb to an initial altitude. Once a safe altitude is reached, the mission execution module within the autonomy engine takes over. A mission is defined as a set of waypoints connected by individual trajectory segments that are calculated using polynomial trajectories following the approach of [Mellinger and Kumar, 2011, Richter et al., 2016]. A trajectory tracking module plays the mission trajectory forward in time by issuing position references to the low-level position controller. During mission execution, a system health observer monitors all critical components of the UAS and issues emergency messages to the autonomy engine to implement fail-safe behaviors (e.g. return-to-home, emergency-landing on low battery or state estimation failure).

After the mission is completed, the UAS returns to the vicinity of the landing station using the recorded GPS position of the take-off location. Our vision-based landing navigation algorithm then detects the landing station AprilTag fiducials with its downfacing camera. Once the landing location is accurately detected, a landing maneuver is executed by first hovering over the landing location and then issuing a trajectory with a fixed descent velocity that passes through the center of the charging pad surface. Touchdown is triggered by detecting zero velocity below a minimum altitude threshold.

4 Navigation

The navigation subsystem is responsible for estimating the UAS state and the landing pad pose in the world frame. The landing pad pose estimation is designed to provide a sufficiently accurate pose estimate to be able to land on a $90\times 90$ cm surface. The vehicle pose estimator is designed to work in an outdoor environment such as a crop field.

4.1 Visual Landing Navigation

Figure 7 details our landing site detection algorithm. The downfacing camera feeds a distorted 752$\times$480 px grayscale image, $\mathcal{I}_{\text{raw}}$, into the image undistorter. The latter removes radial distortion in the image via the ideal fish-eye lens model [Devernay and Faugeras, 2001], producing the un-distorted image $\mathcal{I}_{\text{pin}}$ assumed to be produced by a pinhole camera with calibration matrix $K\in\mathbb{R}^{3\times 3}$. The unmodified AprilTag 2 implementation [APRIL Laboratory, 2016] identifies AprilTag markers in $\mathcal{I}_{\text{pin}}$ and for each tag with ID $i$ produces a homography matrix $H_{i}\in\mathbb{R}^{3\times 3}$. Assuming that at least one tag is detected, a perspective-$n$-point solver [OpenCV, 2017] produces an AprilTag bundle pose measurement $\hat{\bm{p}}_{\text{w}}^{\text{l}}$, $\hat{\bm{q}}_{\text{w}}^{\text{l}}$ of the landing pad in the world frame. This measurement is passed to a recursive least squares (RLS) estimator to output $\tilde{\bm{p}}_{\text{w}}^{\text{l}}$ and $\tilde{\bm{q}}_{\text{w}}^{\text{l,yaw}}$ where the latter is a pure yaw quaternion because we assume the landing pad to be level. The RLS outputs are then used for landing navigation. The algorithm is executed on a single core of an Odroid XU4, except for the AprilTag 2 detector which runs on two cores, and achieves a measurement frequency of approximately 7 Hz as shown in Figure 8.

Estimation of the landing pad pose using a tag bundle has several advantages. First, by providing more points to compute the homography matrix, the landing pad pose measurement becomes more accurate and robust to effects like tag corner detection error. Furthermore, a bundle can be composed of larger and smaller tags such that it is visible from a wide range of altitudes (see Section 8.1.1).

4.1.1 Bundle Calibration

To compute $\hat{\bm{p}}_{\text{w}}^{\text{l}}$ and $\hat{\bm{q}}_{\text{w}}^{\text{l}}$, the transformation matrix from each tag to the landing pad frame must be known. This is shown in Figure 9 and the computation of these transforms is called landing bundle calibration. Because the landing pad pose directly affects the desired UAS landing pose (see Section 4.1.2), calibration is crucial for successful system operation.

Our calibration approach is to place the UAS downfacing camera with the landing bundle visible, forming a rigid body transform triad as shown in Figure 10. For the calibration process, a “master” AprilTag marker is placed at the same position and yaw as desired for the UAS and the downfacing camera respectively during landing (see Figure 11). The unknown calibration transform $T_{\text{lt}_{i}}$ is given by:

[TABLE]

However, every standalone tag pose measurement contains error due to effects from image blur, pixelation, etc. The result of single-measurement computation (1) is therefore not reliable. For this reason, a statistical approach is taken and a sequence of several hundred $\hat{T}_{\text{ct}_{i}}$ and $\hat{T}_{\text{cm}}$ is collected. These measurements are then combined into a geometric median position and a mean quaternion attitude of each tag relative to the landing pad frame. Computational details are provided in [Malyuta, 2018].

4.1.2 Landing Alignment

The UAS can align itself with the landing pad by using landing pad pose estimate $\tilde{\bm{p}}_{\text{w}}^{\text{l}}$ and $\tilde{\bm{q}}_{\text{w}}^{\text{l,yaw}}$. For this purpose, consider the alignment scenario depicted in Figure 11. The desired attitude of the camera is to look straight down at the landing pad frame, hence $\bm{q}_{\text{c}_{\text{des}}}^{\text{l}}$ corresponds to a 180∘ rotation about the $x$-axis. As shown in Figure 11, we have all the elements necessary to compute the desired body pose such that the UAS is center-aligned with the landing frame and the camera is facing down with a yaw implicitly defined by the master tag during the calibration step of Section 4.1.1:

[TABLE]

where $h_{\text{align}}$ is the desired altitude above the landing pad prior to final descent and $\bm{e}_{\text{z}}=(0,0,1)$. Once aligned, the landing autopilot descends the UAS until touchdown as described in Section 7.5.

4.2 Vehicle Pose Estimation

The state estimation subsystem computes the vehicle pose to be used for guidance and control. For this purpose the SSF algorithm [Weiss et al., 2012] is extended to fuse multiple sensors. In particular, we use an IMU, a GPS for position and velocity, a pressure sensor for improved height estimation, and a 3-axis magnetometer for yaw observability during hover and constant velocity flight. We denote by $\{\text{b}\}$, $\{\text{g}\}$, $\{\text{m}\}$ and $\{\text{p}\}$ the body (i.e. IMU), GPS, magnetometer and pressure sensor frames respectively and use the following state vector:

[TABLE]

where $\bm{b}_{\omega}\in\mathbb{R}^{3}$, $\bm{b}_{\text{a}}\in\mathbb{R}^{3}$ and $b_{\text{p}}\in\mathbb{R}$ are the gyroscope, accelerometer and pressure sensor biases and $\bm{m}_{\text{w}}\in\mathbb{R}^{3}$ is a normalized magnetic field vector. The core state portion of (4) together with the raw gyroscope body rate measurement is used for control and guidance. During the landing phase, the landing pad pose estimate $\tilde{\bm{p}}_{\text{w}}^{\text{l}}$ and $\tilde{\bm{q}}_{\text{w}}^{\text{l,yaw}}$ is used to re-localize the landing pad in the world frame. This is necessary due to drift in the GPS sensor during long flights.

5 Guidance

The guidance subsystem is responsible for generating control reference signals with the aim of following a specific trajectory or arriving at a specific destination [Grewal et al., 2007]. This section describes the guidance philosophy and main aspects, while further computational details are provided in [Malyuta, 2018].

The guidance subsystem is designed with the objectives of being computationally lightweight and capable of generating predictable trajectories that are dynamically feasible and that enable the UAS to explore an obstacle-free 3D volume via point-to-point constant velocity segments. Among existing methods that minimize snap [Mellinger and Kumar, 2011, Richter et al., 2016, Burri et al., 2015, de Almeida and Akella, 2017], jerk [Mueller and D’Andrea, 2013, Mueller et al., 2015] and energy [Açıkmeşe and Ploen, 2007, Blackmore et al., 2010], our approach is to compute the trajectories as fully constrained 9-th order polynomials that are smooth up to snap at their endpoints. This avoids online optimization and, in our experience, makes the planned trajectory much more readily predictable than the other approaches.

5.1 Trajectory Segments

A mission is comprised of a sequence of trajectory segments. As shown in Figure 12, there are two main types of trajectories. The first, a transfer trajectory, interpolates all derivatives up to snap smoothly between any two values in a fixed amount of time. The second, a mission trajectory, is a set of constant velocity segments between user-defined waypoints where transitions between different velocities are filleted by transfer trajectories. Figure 13 shows a typical example of a generated waypoint mission, with the UAS body frame sampled along its length. We also provide the ability to hover at any waypoint in the mission trajectory, which occurs at the first and second to last waypoints in Figure 13. It can be seen that the trajectory consists mostly of constant velocity segments. Our approach ensures that these segments between user-defined waypoints are straight, which is desirable for most remote sensing applications.

5.2 Trajectory Sequencer

The trajectory sequencer aggregates individual trajectory segments into a continuous sequence of trajectories. This is implemented as a doubly-linked list where each element stores the trajectory segment, a timestamp and a type which is either single or cyclic where in the former case the trajectory is flown only once and in the latter it is repeated until being aborted by other means such as a low battery charge flag. As described in Figure 14(b), we additionally provide a volatile element which enables executing one-time transfer trajectories to correct for very large trajectory tracking errors.

6 Control

We use a three-stage cascaded control architecture shown in Figure 15 to track the reference trajectories output by the trajectory sequencer. The stages are, from outer- to inner-most loop: translation, attitude and body rate controllers. A cascaded controller has the advantage that inner loops provide faster disturbance rejection and reduce the effect of nonlinearities [Skogestad and Postlethwaite, 2005], which is beneficial for accurate tracking. The controller is executed on the AscTec HLP as shown in Figure 5.

The guidance subsystem forms the outermost loop. It is executed at 20 Hz and outputs a reference position $\bm{p}_{\text{ref}}$, a reference velocity $\bm{v}_{\text{ref}}$, a reference yaw $\psi_{\text{ref}}$ and optionally a feed-forward acceleration $\bm{a}_{\text{ff}}$ and feed-forward body rates $\bm{\omega}_{\text{ff}}$. All quantities are obtainable from our polynomial trajectories via differential flatness theory [Mellinger and Kumar, 2011].

Translation control forms the next inner loop and consists of a pre-filtered two degree of freedom PID controller [Skogestad and Postlethwaite, 2005] which outputs a desired vehicle acceleration $\bm{a}_{\text{ref}}$. Since $\bm{a}_{\text{ref}}$ defines a reference thrust vector, the translation controller may be thought of as a thrust vector calculator. In order to prevent excessive tilt of the UAS, we limit the thrust vector to a $20^{\circ}$ cone half-angle about the vertical axis. Translation control loop bandwidth is between 1 and 2 rad/s. In the event of a loss of state estimation due to communication or sensor failure, a standby emergency landing controller can override the translation controller and land the UAS by inducing a $-2$ m/s vertical velocity in an open-loop fashion similar to [Mueller and D’Andrea, 2012, Lupashin et al., 2014]. This event can occur as a result of serial communication cable failure, for example.

The attitude control forms the next inner loop and tracks $\bm{a}_{\text{ref}}$ and $\psi_{\text{ref}}$ by generating a body rate reference $\bm{\omega}_{\text{ref}}$ and a collective thrust reference $T_{\text{ref}}$ for the body rate controller. Our implementation of the attitude controller is based on the globally asymptotically stable and robust to measurement noise quaternion-based controllers introduced in [Brescianini et al., 2013, Faessler et al., 2015]. The attitude control loop bandwidth is between 5 and 10 rad/s.

The body rate controller forms the inner-most loop of the cascaded flight control system. It tracks $\bm{\omega}_{\text{ref}}$ by computing reference body torques, following the feedback-linearizing control scheme presented in [Faessler et al., 2017]. These along with $T_{\text{ref}}$ are then converted into individual reference motor thrusts $f_{\text{ref}}$ that are mapped to propeller speeds via a full quadratic motor calibration map. Our controller uses a thrust saturation scheme based on [Faessler et al., 2017] which prioritizes roll and pitch torques, which are the most important ones for stability. The body rate control loop bandwidth is approximately 40 rad/s.

7 Autonomy Engine

The autonomy engine subsystem implements the logic for long-duration fully autonomous operation with repeated data acquisition flights.

7.1 Architecture Overview

Figure 2 illustrates the overall concept of operation for repeated data acquisition. Each mission phase is implemented as a stand-alone state machine called an autopilot that executes the logic associated with the phase whenever it is activated via a Robot Operating System (ROS) action interface. To coordinate calls to each autopilot, an additional overarching logic is implemented as a high-level master state machine. The hierarchy of state machines and their interface to other major subsystems is shown in Figure 16. All state machines are executed in parallel threads at a frequency of 20 Hz.

7.2 Master State Machine

The master state machine coordinates calls to the phase-specific autopilots and is illustrated in Figure 17. In contrast to the computation-heavy autopilots, the master performs no computations itself. Its sole responsibility is to initiate event-based state transitions. In the takeoff, mission, landing and emergency landing states the master state machine activates the appropriate autopilot and waits for it to complete. If necessary, the master can also abort each autopilot in order to execute robust behaviors for cases like low battery charge in flight or abnormal motor performance before takeoff. For example, if battery charge anomalously drops to critical levels, the mission and landing actions can be aborted mid-execution to perform an emergency landing action.

7.3 Takeoff Autopilot

The takeoff autopilot takes the UAS from a motors-off state on the charging pad to a hover at a target takeoff altitude. The procedure is illustrated in Figure 18(a). In summary, after successfully validating motor nominal performance, re-initializing the state estimator after the prolonged charging phase and memorizing the current horizontal location in permanent memory, the takeoff autopilot commands a velocity-controlled takeoff that takes it from the initial position on the charging pad to a target altitude.

Two parts of the takeoff procedure are of particular interest. The first is a motor nominal performance check which occurs prior to takeoff. Motors are spun at a low RPM, which is measured via zero crossing detection, and are verified to rotate within 400 RPM of the nominal value. Ten attempts to pass this check are allowed before the takeoff is aborted. Our test flights had instances of this procedure saving the UAS e.g. when an I2C cable to one of the motors dislodged. The next important part of the takeoff is saving the current location to a database in permanent memory. This gives the UAS knowledge of where to return to after the mission and also prevents this information from being lost in case a software reset is required.

7.4 Mission Autopilot

The mission autopilot is responsible for executing the actual data acquisition mission as defined by the user in the form of individual waypoints and hover times. The algorithm is illustrated in Figure 18(b). The mission trajectory is performed either once or is re-flown continuously until an event such as low battery charge requests the UAS to return to the charging pad.

7.5 Landing Autopilot

The landing autopilot safely takes the UAS from a hovering state in the vicinity of the landing pad to a motors-off state on the charging surface. The procedure is illustrated in Figure 18(c). The first action is to check if the landing pad is visible in the downfacing navigation camera image, i.e. if any AprilTag in the bundle is detected. If not, the UAS executes the spiral grid search trajectory until the landing bundle becomes visible. The vision-based landing navigation algorithm (Figure 7) then becomes activated. The estimated yaw is used to align the vehicle attitude such that the camera points towards the visual markers, while the landing position estimate is used to align the vehicle’s lateral position over the center of the charging pad. The vehicle then performs a constant velocity descent until touchdown is detected based on 0.3 m height and 0.1 m/s velocity thresholds:

[TABLE]

7.6 Emergency Lander

The emergency lander brings the UAS to a soft touchdown at its current location and is triggered in response to a critically low battery voltage. Note that this is a different behavior from the emergency landing controller in Figure 15, which handles state estimation failure. The emergency lander logic is illustrated in Figure 18(d). The algorithm descends the UAS with a mild vertical velocity of 0.3 m/s until touchdown is detected based on a 0.1 m/s velocity threshold:

[TABLE]

8 Experimental Results

The system was tested in four configurations of increasing complexity. These were software-in-the-loop (SITL) simulation [Furrer et al., 2016], indoor tethered flights with VICON-based state estimation, outdoor tethered flights with GPS-based state estimation and outdoor free flights. SITL simulation allows for rapid testing and debugging by removing most logistic concerns and by making it easier to identify algorithmic errors due to the absence of real-world noise. Indoor test flights under VICON allow to refine the guidance, control and autonomy subsystems independently of state estimation. Finally, outdoor flights validate that the system works in its real-world environment.

8.1 Vision-based Landing Accuracy

To design an AprilTag marker bundle for localizing the landing pad as explained in Section 4.1, we first analyzed the detection accuracy in a stationary setting of the AprilTag 2 algorithm for single tags at various observation distances using the downfacing camera that is deployed on our UAS (see Figure 4).

Figure 19 shows the standard deviation of the distance estimate for various tag sizes and distances. For a given tag size, the estimation accuracy decreases with increasing distance. Large tag sizes (48 cm, 60 cm) are not detected at distances below 2 m, while small tag sizes (10 cm, 15 cm, 20 cm) cease to be detected above 4 m. For an initial approach height of 4 m, we selected a 48 cm tag for an expected $2\sigma$ (i.e. 2 standard deviation) height estimation error of 4.5 cm and bundled it with three 15 cm tags for increased precision at lower altitudes, resulting in the bundle depicted in Figure 3 and the bottom left of Figure 23.

Figure 20 illustrates an experiment where the quadrotor was flown at various heights above the landing marker bundle shown in Figure 3. The statistics for the error between the height estimate using the AprilTag detector and the ground truth from RTK-GPS are plotted in Figure 21. At the approach height of 4 m the 2$\sigma$ height detection error is 0.32 m. The error decreases to below 0.1 m in the final stage of the descent. We note that this estimation error variance is larger than for the individual tags in Figure 19. The increased error is most likely caused by image blur and latency effects due to the motion of the vehicle and a lever arm effect that amplifies position errors at the desired landing location (the charging pad center) as a result of angular errors in the AprilTag detection. Section 8.1.1 provides a remedy to the lever arm effect.

Figure 22 illustrates the accuracy of autonomous landing in three environments of increasing complexity. First, 200 landings were tested in SITL simulation with a random wind force averaging 30 km/h modeled as a random walk for the wind direction. The $2\sigma$ landing error is 0.09 m (major half-axis) and 0.08 m (minor half-axis). Because simulation uses the ground-truth UAS pose for control, we may conclude that this is roughly the landing error that our control and landing navigation algorithms are able to maintain in severe wind conditions. Next, the UAS performed 27 landings indoors with VICON state estimation in near-optimal conditions. The resulting $2\sigma$ landing error ellipse aggregates all detection and control errors. The lateral $2\sigma$ error is 0.11 m (major half-axis) and the longitudinal $2\sigma$ error is 0.08 m (minor half-axis). We see that the accuracy is similar to simulation, which indicates that real-world uncertainty sources like model inaccuracy and data signal latency degrade landing precision by a similar amount as severe wind conditions with an otherwise ideal system. Finally, 21 landings were performed outdoors using RTK-GPS for ground truth. The lateral $2\sigma$ error is 0.37 m (major half-axis) and the longitudinal $2\sigma$ error is 0.28 m (minor half-axis). While this is well within the charging surface dimensions, the increased landing error is representative of the less accurate outdoor vehicle pose estimate, camera exposure adjustment effects due to light conditions, more complex aerodynamic effects from wind than those modeled in simulation, etc. Because we required the downfacing camera to collect more pad pose measurements in order to initialize the RLS filter from Section 4.1, the outdoor landing trajectories have a visible hover period at the preset 4 m approach altitude in the bottom left of Figure 22.

8.1.1 Optimizing the AprilTag Bundle Layout

We carried out a simulated set of experiments in Gazebo [Koenig and Howard, 2004] to search for an AprilTag bundle geometry which minimizes the landing pad position measurement error. While many factors affect the AprilTag position measurement error, such as the camera-tag off-axis angle [Olson, 2011, Wang and Olson, 2016], our investigation considers only the camera-tag distance while keeping the off-axis angle zero. This choice is appropriate for our use case, since the dominant variable that changes during the landing phase is the quadrotor height and the camera is facing approximately straight down at the AprilTag bundle. Furthermore, the simulation environment allows to isolate the effect of bundle geometry since other variables (e.g. illumination, lens focus, etc.) are easily controlled for. Thus, bundle geometry and camera-tag distance are the only independent variables. Our simulation uses the downfacing camera parameters from Section 4.1, i.e. a 752$\times$480 px resolution and the same calibration matrix $K\in\mathbb{R}^{3\times 3}$.

Figure 23 shows the tag bundle geometries that were tested. Note that the first two rows of bundles in Figure 23 are impractical to land on since they would obscure the charging surface. The bundles in the third row are therefore placed around the charging area.

Figure 24 shows the output of our simulation for the height detection noise variance. We observe that a larger tag decreases measurement noise (as seen for the top row single-tag bundles of Figure 23). Importantly, the detection distance of larger tags is larger because they make a bigger pixel footprint in the image. Measurement noise for bundles in row 2 of Figure 23 is smaller than for the single-tag bundles, however the 5- and 9-tag bundles do not appear to reduce noise with respect to the 3-tag bundle. By using both large and small tags, the landing bundle we used in real-world experiments is able to be detected from large distances of up to 14 m as well as from small distances prior to touchdown. By offsetting the tags to one side of the charging station, however, the detection accuracy is reduced by the aforementioned lever arm effect. An improved bundle, with tags distributed symmetrically about the target landing location, shows better detection performance. However, it is a less practical solution as it takes up a lot more space.

8.2 Autonomy Engine

Figure 25 shows the simultaneous operation of the master state machine and the four autopilots for takeoff, mission, landing and emergency landing in four simulated modes of operation. The first nominal flight is the one illustrated in Figure 2. In the second flight, a low battery charge aborts the mission and the UAS returns to the charging pad. In the third flight, the UAS spends some time executing the spiral grid search trajectory because the landing marker bundle is initially not visible. In the fourth and final flight, a critically low battery charge forces the UAS to land in-place during the mission because it is now deemed too dangerous to make a safe return to the charging pad. This final scenario is an important robustness mode which reduces the risk of the UAS crashing due to a battery malfunction.

8.3 Long-Duration Autonomy and Recharging

We tested the system during a set of indoor and outdoor deployments. For component verification, we initially performed indoor experiments in VICON to verify recharging only (11 h experiment with 16 flights) and then the full system (10.6 h with 48 flights). We then proceeded to test the system outdoors with a mission profile focused on maximizing the number of performed precision landings. This is because precision landing is the most difficult mission phase to execute when taking the UAS from the high-precision indoor motion capture environment to a much lower-precision camera, IMU and GPS-based navigation solution outdoors (see Section 4). The outdoor experiment was hence a mission cycle consisting of taking off, flying to several GPS-based waypoints for the duration of about 1 minute and then returning to the charging pad for landing and recharging.

Among multiple outdoor deployments, our system achieved a 6 h tethered and a 4 h free flight fully autonomous operation during which 22 flights were performed. Figure 27 illustrates the state transitions of the master state machine during the free flight outdoor experiment and the associated battery voltage reflecting the individual charging cycles. Figure 28 shows a breakdown of the times spent in the individual mission phases, where it is seen that the UAS spent about $10~{}\%$ of the time collecting data.

Although the objective of the outdoor experiment was repeatable precision landing and not data collection, an on-board thermal camera monitored ground surface temperatures during the course of the experiment. Figure 26 shows an example mission data product monitoring the surface temperature of a rock formation at different times of day.

8.3.1 Lessons Learned

Outdoor operation posed several new challenges that are absent in an indoor environment. First, dirt and dust collecting on the charging pad has at several points prevented a proper contact with the charging pad, hence preventing recharging. Next, camera exposure becomes a crucial factor in outdoor operation where light conditions change throughout the day, sometimes very quickly as clouds block sunlight. As a result, automatic exposure control is important. Improvements can be made by adapting exposure control to the expected brightness level of the observed landing target, or by using a different technology like an event-based camera which has been shown to be less susceptible to exposure effects [Vidal et al., 2018].

We also found that GPS accuracy is highly sensitive to electromagnetic interference from nearby emitters such as Wi-Fi dongle, on-board computer and USB connections [Davuluri and Chen, 2013]. A careful hardware design must hence be performed to maximize outdoor state estimation accuracy and to prevent sudden GPS degradation due to signal interference in mid-flight. From a software perspective, we found the AscTec MAV framework [Achtelik et al., 2017], which interfaces the AscTec autopilot board in Figure 5 to ROS, to frequently crash after long periods of operation.

From an operational standpoint, the flight time to charging time ratio in Figure 28 is rather low. Similar ratios have been observed before [Valenti et al., 2007]. The main limiting factors here were the weight of the on-board power supply, which decreased flight times, and a need to guarantee safe LiPo battery operation, which limited the charging current and increased the charging times. A possible solution would be to re-engineer the battery/charging pad system or to adapt a battery swapping approach. Our software framework can easily accommodate this change.

In summary, long-term platform autonomy outdoors is hindered by minor issues like dust accumulation, camera exposure, GPS accuracy due to signal interference and software stability. We believe that each challenge can be solved by further hardware and software engineering, though such fine tuning is beyond the scope of this initial research and development. Nevertheless, due to its modular design, we believe that the autonomy engine presented herein can readily accommodate any such modification.

9 Future Work

Several future improvements are conceivable for our system. For navigation, a rigorous analysis of latency in the data signals is necessary to improve platform accuracy. For precision landing, AprilTag bundle pose measurement frequency may be increased from the current 7 Hz to gather more data and therefore to improve the estimate of the landing pad pose. To similar effect and to increase the maximum landing approach height, the downfacing camera resolution may be increased (e.g. [GoodRobots, 2019] report tag detection from a 32 m altitude using 1080p images).

Several improvements are possible for guidance and control. Current trajectories are simple fully constrained polynomials. To extend flight time, energy-minimizing optimal trajectories may be generated by solving an optimization problem. A potential avenue is to adapt existing research on using convex optimization to plan reliable minimum-fuel trajectories for planetary landers [Açıkmeşe and Ploen, 2007]. There is also more direct research for multirotor minimum-energy and minimum-time trajectory generation [Vicencio et al., 2015, Morbidi et al., 2016, Mueller et al., 2015]. On the control side, our experience shows that closed-loop motor RPM control can benefit trajectory tracking accuracy as it effectively renders the motor thrust characteristic independent of battery voltage. This makes the control system less dependent on integrators for error correction.

Several improvements to the autonomy engine implementation are possible. More expandable behavior can likely be achieved by porting the existing state machines to a framework such as SMC [Rapp et al., 2017] which follows the open-closed design pattern of keeping software entities open for extension but closed for modification. In practice, this means that new autonomy engine states may be added without modifying source code for existing states. Furthermore, the current implementation is not robust to individual ROS node crashes. This creates single points of failure such as the AscTec HLP interface which, if it fails, will prevent the state estimate from being sent to the HLP and will trigger the emergency landing controller. A potential solution is to implement a highly reliable supervisory software that is able to restart failed ROS nodes, or the entire ROS network if necessary.

10 Conclusion

In this work, we presented an architecture for a long-duration fully autonomous UAS for remote-sensing data acquisition missions. Such a system is crucial in enabling precision agriculture and other applications that require a long-term remote-sensing capability such as environmental monitoring and surveillance. Our system’s two major components are a fully autonomous aerial vehicle and a landing station which integrates a charging pad. The aerial vehicle carries our autonomy software which enables it to perpetually execute a user-defined mission and to downlink collected data to a base station computer. We were able to operate this system in several environments of increasing complexity, culminating in a fully autonomous 4 h outdoor operation in which 10 % of the time was spent flying to collect data. This represents to the best of our knowledge the first research publication on a UAS system that is capable of long-duration full autonomy outdoors.

Acknowledgments

This research was carried out at the Jet Propulsion Laboratory, California Institute of Technology, under a contract with the National Aeronautics and Space Administration. The authors would like to thank Stephan Weiss for his inputs on vehicle pose estimation in Section 4.2 and our JPL collaborators Darren Drewry and Debsunder Dutta for providing the NDVI data illustrated in Section 1.1.

Copyright 2018, California Institute of Technology. U.S. Government sponsorship acknowledged.