Visual-Inertial Teach and Repeat for Aerial Inspection

TL;DR

This paper introduces a system enabling human operators to teach aerial inspection tasks to drones via demonstration, utilizing visual-inertial mapping for robust operation in GPS-denied environments, with real-time multi-robot tracking capabilities.

Contribution

The novel system allows easy teaching of inspection paths to MAVs using a handheld device and employs a visual-inertial framework for accurate localization without GPS.

Findings

Successful autonomous inspection path following

Robust operation in GPS-denied environments

Real-time multi-robot pose tracking

Abstract

Industrial facilities often require periodic visual inspections of key installations. Examining these points of interest is time consuming, potentially hazardous or require special equipment to reach. MAVs are ideal platforms to automate this expensive and tedious task. In this work we present a novel system that enables a human operator to teach a visual inspection task to an autonomous aerial vehicle by simply demonstrating the task using a handheld device. To enable robust operation in confined, GPS-denied environments, the system employs the Google Tango visual-inertial mapping framework as the only source of pose estimates. In a first step the operator records the desired inspection path and defines the inspection points. The mapping framework then computes a feature-based localization map, which is shared with the robot. After take-off, the robot estimates its pose based on this map and plans a smooth trajectory through the way points defined by the operator. Furthermore, the system is able to track the poses of other robots or the operator, localized in the same map, and follow them in real-time while keeping a safe distance.

Full text

11footnotetext: Autonomous Systems Lab, ETH Zurich, Switzerland.

{marius.fehr, thomas.schneider, marcin.dymczyk}@mavt.ethz.ch, rsiegwart@ethz.ch22footnotetext: Google Inc.: jsturm@google.com††This reasearch was funded by Google Tango.

I INTRODUCTION

Industrial facilities such as refineries, power and heating plants are required to operate at peak efficiency over extended periods of time. Interruptions or failures due to wear and tear on the components of such installations are to be avoided at all cost, hence, regular inspections and revisions of key components are imperative. However, inspecting these points of interest is time consuming and therefore expensive, especially if the installations need to be shut down due to potential or actual hazards to the technicians. Other components might just simply be hard to reach and require special equipment. Automating these inspection tasks using robots, such as MAVs , is a very cost-efficient solution, which can decrease down-times and most importantly keep the human work force out of harms way.

However, autonomous MAVs require exact pose estimates to safely navigate in these potentially confined and GPS-denied environments. That’s why much of the industrial inspection research focuses either on the inspection task itself by providing change/damage detection algorithms [2, 3] or additional safety/navigation mechanisms to support the human pilot [4, 5, 6]. With the advent of robust Visual-Inertial Odometry (VIO) pipelines and mapping framework, such as maplab [7], ORB-SLAM2 [8], OKVIS [9], ROVIO [10] and the visual-inertial navigation system presented in [11] autonomous robots are able to perform more challenging tasks. Without the need for external infrastructure, like cameras, markers or beacons, these systems allow the robots to follow waypoints provided by the operator in visually challenging environments [12] or fully reconstruct the environment for the purpose of collision avoidance [13].

In this work we present a complete Teach&Repeat (T&R) system based on the Google Tango mapping framework [1] that allows a non-expert human operator to simply demonstrate an industrial inspection task to the robot by simply walking and pointing using a hand-held tablet. This allows the automation of tedious and potentially hazardous inspection routines in no more time than it takes to manually perform a single such task. In case the inspection point is out of reach for the human operator, the inspection task can also be taught by manually piloting the MAV . The on-board VIO allows the operator to navigate precisely and safely in position control mode, considerably lowering the need for extensive pilot training.

The contributions of this work are as follows:

• •

We present a novel, robust visual-inertial-based T&R system for industrial inspection.

• •

We demonstrate the capabilities of the system in a challenging real-world industrial setting.

II SYSTEM

In this section we will introduce the architecture and components of the presented T&R system. Fig. 2 provides an overview of the components and data flow of the proposed inspection system which is comprised of a teacher and an agent. The teacher refers to a human operator either carrying a tablet or piloting an MAV . The agent on the other hand is a fully autonomous MAV . The system can be used in two different modes: In live-mode the MAV will follow the operators poses in real-time, while keeping a safe distance in case it catches up. In T&R* -mode* the MAV will record the operators poses and inspection points while on the ground and is able to then later execute the observed task as often as required. In both modes the communication between teacher and agent was maintained over WiFi using UDP connection.

The remaining communication between the components as well as the components themselves are implemented based on the Robot Operating System (ROS)  [14].

**State estimation: ** In order to be able to perfectly reenact the inspection task demonstrated by the teacher, both teacher and agent need to estimate their pose with respect to a common global frame. This is achieved by means of robust VIO and localization based on binary feature descriptors. In order to create a localization map the initial map from odometry is post-processed using appearance-based loop closure and Visual Inertial Weighted Least-Squares (VIWLS) optimization (See Fig. 3). Using this common localization map 6-DOF poses taught by the operator can be precisely mapped to the corresponding $[x,y,z,yaw]$ waypoints required by the MAV . On the MAV the localized 6-DOF pose provided by the odometry is furthermore fused with the on-board Inertial Measurement Unit (IMU) for additional robustness using a Multi Sensor Fusion (MSF) based on [15].

T&R** planner: ** This global planner tracks the poses of the teacher and maintains a key-framed recording of its trajectory including the desired inspection points. Based on the MAVs position the planner will select a sequence of key-frames along the teachers trajectory. There are two ways such a sequence can end, either in a normal key-frame, limited by a predefined maximum length or in an inspection point. The planner will furthermore compute the desired velocity at the end of this sequence. If the final waypoint is not the end of the trajectory and not equal to an inspection point this velocity is set to the maximum allowed velocity, otherwise the planner will set the velocity to zero.

**Safety: ** The sequence of waypoints and current agent position is continuously checked for collision against the safety spheres around both teacher and other agents. If a sequence intersects with such a safety sphere, it is cut short and the terminal velocity set to zero, such that the agent comes to a full stop at a safe distance. In case the agents own safety sphere intersects with another sphere, an emergency stop signal is sent directly to the controller, ordering an immediate stop of the MAV . It is important to note that aside from passive collision avoidance based on the known poses of the teacher and other agents, the system does not have any collision avoidance capabilities. It therefore assumes a static environment with known dynamic obstacles.

**Local planner: ** The waypoints computed by the T&R planner are then used by the local planner to compute a smooth and dynamically feasible trajectory for the MAV using polynomial trajectory optimization similar to [16].

**Control: ** Finally, the polynomial trajectory is sent to the non-linear Model Predictive Controller (MPC)  [17], which in turn computes the desired thrust and attitude signals for the MAV interface.

III RESULTS

We demonstrate the capabilities of the proposed system in the following video: https://goo.gl/V3TM6N. The experiments were conducted on an AscTec Neo hexacopter, equipped with a visual-inertial stereo sensor and an Intel NUC, i7 on-board computer. Prior to the experiments the operating environment was recorded using a Google Tango tablet. The data was then downloaded to a external computer, loop-closed and bundle-adjusted and finally converted to a localization map using the Google Tango framework [1]. The resulting localization map can be seen in Fig. 3. At the beginning of the experiment the localization map was distributed among the teacher and the agent, i.e. the tablet and the MAV . All subsequent processing was executed exclusively on-board the tablet and the MAV . Localized to a common frame of reference, the inspection task is defined by the human operator using a hand-held Google Tango tablet. Inspection points are automatically inserted if the operator remains motionless for 2 seconds. The resulting inspection task can be seen in Fig. 4, where the MAV has just started following the taught trajectory. Furthermore, Fig. 4 shows the inspected installations as seen from the MAV after it safely and reliably completed the desired inspection task.

IV CONCLUSION

This work presents a novel system for autonomous industrial inspection using MAVs , where the inspection task can be taught in an intuitive and safe manner by a human operator holding a tablet or piloting an MAV . In a GPS-denied environment, robust visual-inertial mapping techniques are used to localize teacher and agent in a common frame of reference, without the need for external infrastructure, such as markers or beacons. To demonstrate the capabilities of the proposed system, we conducted a challenging real-world experiment in an industrial environment. The MAV followed the inspection plan safely and precisely and succeeded in observing all inspection points.

ACKNOWLEDGMENT

The presented experiments received support from members of the Autonomous Systems Lab and Google Tango, most importantly: Michael Burri, Helen Oleynikova, Zachary Taylor, Fabian Blöchliger, Mingyang Li, Ivan Dryanovski, Simon Lynen, and Konstantine Tsotsos.