Visual-Thermal Landmarks and Inertial Fusion for Navigation in Degraded Visual Environments

TL;DR

This paper introduces a multi-sensor fusion approach combining visible, thermal, and inertial data to enable reliable autonomous navigation for aerial robots in GPS-denied and visually degraded environments, such as industrial and subterranean settings.

Contribution

It presents a novel fusion algorithm that leverages thermal and inertial sensors for robust odometry in challenging conditions, improving over traditional visible-only methods.

Findings

Successful navigation in industrial environments with obscurants

Effective feature extraction using spatial entropy metrics

Enhanced robustness through inertial sensor integration

Abstract

With an ever-widening domain of aerial robotic applications, including many mission critical tasks such as disaster response operations, search and rescue missions and infrastructure inspections taking place in GPS-denied environments, the need for reliable autonomous operation of aerial robots has become crucial. Operating in GPS-denied areas aerial robots rely on a multitude of sensors to localize and navigate. Visible spectrum cameras are the most commonly used sensors due to their low cost and weight. However, in environments that are visually-degraded such as in conditions of poor illumination, low texture, or presence of obscurants including fog, smoke and dust, the reliability of visible light cameras deteriorates significantly. Nevertheless, maintaining reliable robot navigation in such conditions is essential. In contrast to visible light cameras, thermal cameras offer visibility in the infrared spectrum and can be used in a complementary manner with visible spectrum cameras for robot localization and navigation tasks, without paying the significant weight and power penalty typically associated with carrying other sensors. Exploiting this fact, in this work we present a multi-sensor fusion algorithm for reliable odometry estimation in GPS-denied and degraded visual environments. The proposed method utilizes information from both the visible and thermal spectra for landmark selection and prioritizes feature extraction from informative image regions based on a metric over spatial entropy. Furthermore, inertial sensing cues are integrated to improve the robustness of the odometry estimation process. To verify our solution, a set of challenging experiments were conducted inside a) an obscurant filed machine shop-like industrial environment, as well as b) a dark subterranean mine in the presence of heavy airborne dust.

Equations

$$e_{image}=-\sum _i^N{\vec{p}}_i\cdot \log ({\vec{p}}_i)$$

$$e_{region}=-\sum _i^{N_R}(\vec{\omega }\odot {\vec{p}}_i)\cdot \log ({\vec{p}}_i)$$

$$\mathbf{x}=[\underset{\text{robot states}}{\underbrace{\mathbf{r}\mathbf{q}\mathbf{\upsilon }{\mathbf{b}}_f{\mathbf{b}}_{\omega }}}|\underset{\text{landmark states}}{\underbrace{{\mathbf{\mu }}_0,\cdots {\mathbf{\mu }}_J{\rho }_0\cdots {\rho }_J}}{]}^{\top }$$

$$D_{Optimality}=\exp (\log (\det (\Sigma {)}^{1/l}))$$

Full text

1. 1 Introduction

2. 2 Related Work

3. 3 Proposed Method

4. 4 Experimental Evaluation

5. 4.1 System Overview

6. 4.2 Indoor Experiments

7. 4.3 Subterranean Mine Experiment

8. 5 Conclusions

1 Introduction

Over the past decade aerial robots have seen an unprecedented expansion in the areas of their application as they provide an agile and flexible platform to perform human-reserved tasks autonomously, while minimizing the risk to human life and lowering operational costs. As a natural progression in this expansive role of aerial robots, they are now utilized in applications such as infrastructure inspection [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] and search and rescue missions [11, 12, 13, 14, 15]. These applications require robots to navigate through a variety of environments, while often encountering GPS–denied and visually degraded conditions making reliable pose estimation for the robot a major challenge. In the absence of external pose estimation such as GPS, which is common for indoor search and rescue operations, as well as inspection tasks of tunnel and mine environments, aerial robots rely on their on-board sensors to estimate odometry as they navigate an environment and perform their assigned tasks. For small unmanned aerial robots, visual spectrum camera sensors remain a popular choice as they provide a light-weight, power-efficient and low-cost sensor to enable odometry estimation as compared to 3D LiDAR and RADAR sensors. However, in environments with poor–illumination and low texture, image data from visual cameras become degraded and cannot be reliably used for pose estimation. Furthermore, the visual image quality suffers in the presence of visual obscurants such as smoke, fog and dust. In contrast to the limitations of visible spectrum cameras, thermal cameras operating in the Long Wave Infrared (LWIR) spectrum are not affected by conditions of poor-illumination and the presence of most visual obscurants, while maintaining the desired attributes of being light-weight and power-efficient. However, in general thermal images offer lower contrast in comparison to visual images especially when imaging areas containing objects of similar temperature and emissivity values and therefore require techniques such as histogram equalization to improve contrast [16].

Based on the discussion above, for aerial robots to reliably estimate their pose in a variety of challenging environments a multi–spectral odometry estimation approach, utilizing visual and thermal information in a complementary manner, can offer robustness when one sensing modality becomes degraded. In addition, such an approach is well suited for applications where aerial robots have to move between areas with different cases of sensor degradation, e.g. during search and rescue missions aerial robots might need to move from well–lit but thermally flat areas to smoke–filled areas and vice versa. Similarly, some obscurants might be temporarily introduced due to the operation of the aerial robot itself over different terrains, e.g. during underground mine inspection tasks, an aerial robot creates airborne dust due to the introduced air turbulence in often dry and dusty environments.

Motivated by the aforementioned challenges, in this paper we present an approach to use entropy information in order to selectively utilize visual and thermal image information for the selection of landmarks for the pose estimation of aerial robots. These landmarks are then further fused with inertial measurements provided from an Inertial Measurement Unit (IMU) using an Extended Kalman Filter (EKF) framework. To demonstrate the real-time and on-board performance of our approach in degraded visual environments a set of experiments are conducted using an aerial robot, first in a fog-filled indoor environment. We demonstrate that the aerial robot can maintain a reliable estimate of its pose in the presence of fog, as well as while transitioning from well-lit to poorly-illuminated parts of the environment. Furthermore, we demonstrate the real world performance of our algorithm by estimating the pose of the aerial robot inside a subterranean mine environment in the absence of external illumination and in the presence of heavy airborne dust. A video of the conducted experiments can be found at https://youtu.be/aqZugneeCxc. Figure 1 presents an instance of an underground mine exploration mission.

The remainder of the paper is structured as follows: Section 2 provides an overview of the related work, while Section 3 details our proposed method. The experimental evaluation studies are presented in Section 4. Finally, conclusions are drawn in Section 5.

2 Related Work

Estimation of vehicle pose in Degraded Visual Environments (DVE) has been a widely studied topic in the context of navigation and landing of helicopters especially in conditions of darkness and in the presence of heavy airborne dust. Solutions proposed for larger aerial vehicles are usually based on LiDAR [17] or RADAR [18] systems given the higher payload capacity of such vehicles and how cost factors are considered. However, given the smaller payload capacity of Micro Aerial Vehicles (MAVs), LiDAR and especially RADAR system integration comes with high power and operational time penalties. Considering the case of lightweight sensing solutions carried on-board aerial robots, most solutions emphasize on visual–inertial localization and only more recently the community has considered the potential of thermal vision. The work in [19] showed a hand–held setup consisting of an RGBD and a thermal camera for thermographic mapping of objects. However, the underlying odometry estimation relies purely on RGBD data. Similarly, in [20] a hand–held system consisting of visible and thermal cameras along with an IMU was presented. Authors demonstrated the feasibility of thermal and IMU only odometry estimation, however they also showed the limitations of such a system in areas with very low thermal gradients, while also showing visual odometry to work in the same environment and concluded by suggesting the usage of a multi–spectral approach for odometry estimation. In [21] authors demonstrate odometry estimation of a car using a monocular thermal camera by extending visual odometry approaches but discuss the necessity of knowing the height of camera from the ground at all times to estimate the scale correctly. The authors also discuss the lack of contrast in thermal images and suggest improving the contrast in thermal images, as well as emphasizing the need for the fusion of thermal imagery with other sensing modalities to make odometry estimation more robust. A stereo thermal camera system is proposed in [22] for the estimation of odometry with correct scale. The authors demonstrate the pose estimation of an aerial robot with this system in outdoor settings but discuss the need of integration of proprioceptive sensors such as an IMU to make the overall odometry solution robust.

Following the direction of a multi–spectral odometry estimation solution that utilizes both visible and thermal information, the authors in [23] propose detecting features in both visual and thermal images for odometry estimation. They demonstrate the feasibility of their approach by estimating odometry under different lighting conditions in outdoor settings. Their method proposes separate detection of features in visual and thermal images but extraction of feature descriptors in both domains for each feature point for robust feature matching. However, as discussed in [24] the direct application of feature detection and descriptor matching methods designed for visual images on thermal camera data results in poor feature matching performance and reduced accuracy of the estimated odometry. Echoing the same argument, the authors in [25] propose an edge–histogram based descriptor for multi–spectral feature matching on a visual and thermal camera stereo system. They demonstrate the application of their method by estimating the odometry of a car navigating through an urban environment. However, as discussed in the experimental section of the paper, the proposed descriptor shows inferior feature matching performance when applied to visual images and is replaced with typical visual methods during experiments. Similarly, their odometry estimation approach integrates pose priors provided by GPS making their solution unsuitable for GPS–denied environments. In [26], authors estimate the pose of an unmanned ground vehicle using visual and thermal cameras in the presence of a variety of obscurants in outdoors settings. Features are selectively chosen from areas of visual and thermal images which are not degraded by the presence of obscurants and are used to estimate the robot pose. Scale is calculated independently by using velocity information provided by the IMU. The authors also discuss poor feature matching performance on thermal images, as well as in the presence of obscurants, and suggest the application of methods such as RANSAC, and its variations, to prune incorrect feature matches for improving the reliability of odometry estimation.

Motivated by progress in the literature and the need to address the identified limitations of the current state-of-the-art, in this paper we propose a multi–modal, multi–spectral approach on visual/thermal camera and IMU fusion. The proposed solution utilizes spatial entropy to prioritize certain visual and thermal image regions for feature selection and fuses IMU cues in an EKF fashion. We demonstrate the strengths of our solution in GPS-denied, dark and obscurants-filled environments.

3 Proposed Method

To enable reliable odometry estimation for aerial robots in degraded visual environments (DVEs) using visible and thermal cameras, the selection of interest points (or features) must be made carefully. As discussed in Section 2, the application of visual feature detection and matching methods on images in the presence of obscurants can lead to poor estimation of odometry because of feature mismatching. This is due to the fact that feature detection and matching methods operate over different neighborhood sizes in an image with the detection of features being performed first by evaluating the gradient information over a small neighborhood at each pixel location in the image. Once a set of features has been selected, a feature descriptor in feature–based methods, or an intensity patch in direct intensity methods, is then extracted to encode information over a larger neighborhood around each feature location for matching and tracking in subsequent frames. As the neighborhood sizes for evaluation are different in the detection and matching operations, a feature may be selected which is associated with a non-informative larger neighborhood for matching operations. Such an incorrect selection of features can easily be made in the presence of obscurants such as fog and smoke, which unevenly decrease the quality of gradient information in an image [26]. Furthermore, obscurants such as airborne dust can also temporarily introduce strong local gradients which can lead to the selection of non-repeatable features. This may lead to tracking failure or incorrect estimation of confidence metrics for an odometry solution. An example of temporary strong local gradients introduced by dust particles is shown in Figure 2, where dust speckles introduce strong local gradients against a dark background and can be incorrectly selected as features.

In our approach we associate the local feature selection process to larger image neighborhoods to make sure that the selected features have corresponding informative neighborhoods in order to ensure better feature tracking performance. For this purpose we first measure the amount of information contained in an image which has traditionally been done by using the Shannon Entropy criterion, given as:

[TABLE]

where $e_{image}$ is the entropy measure of the image and $\vec{p}$ is the probability distribution vector of image intensity values over all image pixels $N$ in the image. However, we are interested in measuring the amount of spatial structure present in an image as features typically lie at edge or corner points of objects in the frame. Hence, instead of using direct image intensity values for entropy calculation we make use of the edge-filtered gradient image as it better encodes the underlying spatial structure present in an image as shown in [27]. Moreover, instead of calculating a global metric for an image we divide the image into $R\times R$ regions and calculate the spatial entropy of each region. This division allows us to better preserve local edges in the gradient image which otherwise might have been lost when subjected to a global threshold. Furthermore, although spatial entropy can provide a measure of the structural information available in an image region, it still remains context–free, as a region with an equal amount of very small and very large gradients produces the same spatial entropy measure as a region with a uniform distribution of gradient values as detailed in [28]. Since we are interested in selecting features from regions of the image with a more uniform distribution of gradients, we further introduce a weight vector to weigh our entropy vector using a Gaussian distribution. Our entropy equation for an image region can now be written as:

[TABLE]

where $e_{region}$ is the entropy measure of the region, $\vec{\omega}$ is the Gaussian weight vector, $\odot$ represents the Hadamard product between two vectors and $\vec{p}_{i}$ represents the probability distribution vector of gradient magnitude values over all pixels $N_{\mathbf{R}}$ in a certain image region $\mathbf{R}$. Once the region–wise spatial entropy of the image has been calculated we compute the mean spatial entropy among all the regions and mark regions having an entropy measure above the mean as candidate regions from which features can be selected. Furthermore, in order to account for the dynamic nature of obscurants, we track the spatial entropy of a region over a temporal window and allow the selection of features from a region only if the region remains informative over a select number of frames. Figure 3 shows the application of the combined spatial and temporal quality metric on visible and thermal images, where green boxes indicate candidate regions for feature selection, red boxes show rejected regions which were labeled as candidate regions in the current frame but rejected due temporally inconsistent quality.

Once features have been selected from the informative regions of visible and thermal images, they are fused with inertial measurements using an Extended Kalman Filter (EKF) for the odometry estimation of the aerial robot. A schematic diagram of our approach is shown in Figure 4.

We use an EKF formulation similar to the one proposed in [29] and track features as part of our filter state by modeling them as $3\textrm{D}$ landmarks using a $2\textrm{D}$ bearing vector $\boldsymbol{\mu}$, parameterized with azimuth and elevation angles, and an inverse depth parameter $\boldsymbol{\rho}$. The filter state propagation is performed using proper acceleration $\hat{\mathbf{f}}$ and rotational rate measurements $\hat{\boldsymbol{\omega}}$ provided by the IMU. Propagating the filter state with IMU readings allows the method to predict the feature locations between successive frames, reducing the search space for feature matching in the filter update step and negates the need for a separate feature mismatch pruning step. Feature matching is performed by minimizing the photometric error of a neighborhood patch at each feature’s predicted location. In our formulation, four coordinate frames namely, the IMU fixed coordinate frame ($\pazocal{B}$), the visible camera fixed frame $\pazocal{V}$, the thermal camera fixed frame $\pazocal{T}$, and the world inertial frame $\pazocal{W}$, are used. It should be noted that we do not separately reserve features in the state vector for either visible or thermal images and choose the best available feature across spectra when selection of a new feature is needed. Permitting interchangeable features to be tracked as part of the filter state allows the method to keep the filter state vector small and therefore remain computationally efficient. The employed state vector of our EKF formulation takes the form:

[TABLE]

where $\mathbf{r}$ and $\boldsymbol{\upsilon}$ are the position and velocity of the IMU respectively, expressed in the $\pazocal{B}$ coordinate frame, $\mathbf{q}$ is the IMU attitude from the $\pazocal{B}$ to the $\pazocal{W}$ coordinate frames, $\mathbf{b}_{f}$ and $\mathbf{b}_{\omega}$ represent the additive accelerometer and gyroscope biases respectively expressed in the $\pazocal{B}$ coordinate frame, while $\boldsymbol{\mu}_{j}$ is the bearing vector to the feature $j$ expressed in the $\pazocal{V}$ or the $\pazocal{T}$ coordinate frame and $\rho_{j}$ is the inverse depth parameter of the $j^{th}$ feature such that the feature distance $d_{j}$ is $d(\rho_{j})=1/\rho_{j}$.

4 Experimental Evaluation

To evaluate the performance of our proposed algorithm, a set of experimental studies were conducted using an aerial robot in a) an indoor machine shop environment subject to the presence of dense fog and partial darkness, as well as b) a subterranean mine environment in the presence of heavy airborne dust. The robotic system used and the experiments conducted are detailed below. A video of the relevant experiments can be found at https://youtu.be/aqZugneeCxc

4.1 System Overview

For the purposes of experimental evaluation, a DJI Matrice 100 quadrotor equipped with the sensing suite proposed in this paper and an on-board computer was used. A FLIR Tau 2 thermal camera was mounted on the robot to provide thermal images of $640\times 512$ resolution at $30$ frames per second. Two Pointgrey Chameleon 3 visible light CCD cameras, with shutter–synchronized LEDs, provided visual images of $644\times 482$ resolution at $20$ frames per second. Only one of the visual cameras alongside the thermal camera was used during the experimental evaluations of this work. A VectorNav VN–100 IMU was employed to provide inertial measurements at an update rate of $200\textrm{Hz}$. The intrinsic calibration parameters of the visual and thermal cameras, as well the extrinsic calibration parameters for the IMU and the cameras were estimated using the work in [30] and our custom designed thermal checkerboard pattern [31]. An Intel NUC–i7 computer (NUC7i7BNH) was carried on-board the robot to perform all high-level processing tasks. The estimated odometry from our proposed method is fused with inertial measurements received from the low–level attitude controlling autopilot of the robot using the the work of [32], in order to enable smooth and reliable position control using a Linear Model Predictive Control strategy based on the previous work in [33]. All algorithms were implemented as Robot Operating System (ROS) nodes and run in real-time fully on-board the robot with the odometry estimation operation performing at an update rate of $20\textrm{Hz}$. Figure 5 provides the system overview of the robotic system used.

4.2 Indoor Experiments

To evaluate the real–time on-board performance of the proposed approach on an aerial robot, flight tests were conducted in an obscurant-filled environment. A fog generator was used to fill a machine shop resembling industrial environment with fog to serve as an obscurant during the experiment. Furthermore, lights were turned off in a section of the environment to simulate partial darkness and to demonstrate the robustness of our approach under varying light conditions and in the presence of obscurants during the same experiment. It should be noted that no on-board illumination was used in these experiments. The estimated robot trajectory during flight, from take–off to landing, is shown in Figure 6.

To understand the effect of our feature selection approach on the track–ability of selected features inserted into the filter sate, we conducted 10 trials with and without using our feature selection region mask on the collected data-sets during the indoor experiments. The results of these trials are shown in Figure 7. The plot demonstrates the benefit of our feature selection approach as the median number of features inserted in the filter state is smaller, indicating that the features selected using our proposed approach are tracked for a greater number of frames. Furthermore, to verify that our feature selection strategy provides a statistically significant difference in the selection of features we performed one–way Analysis of Variance (ANOVA) test on the population of features selected with and without the application of our approach. The test resulted in $p\text{-}value<0.007185$, indicating that the two populations are statistically different from each other and verifies the effect and impact of our proposed feature selection strategy.

Likewise, to understand the effect of our proposed feature selection strategy on the quality of the robot pose estimates, we analyze the evolution of the associated covariance of the robot pose estimates over time. As recommended in [34], we employ the D-Optimality criterion as a metric for measuring uncertainty of our robot pose estimates, with a lower D-Optimality value indicating a lower uncertainty in the robot pose estimate. This takes the form:

[TABLE]

where $\Sigma$ is the robot pose covariance matrix with dimensions $l\times l$. The evolution of the D-Optimality criterion for the robot pose estimates over the length of the mission is shown in Figure 8. It can be noted that the application of our feature selection mask attenuates the accumulation of uncertainty in robot pose estimates, as features that can be tracked for more successive frames lead to better convergence of filter uncertainty estimates.

4.3 Subterranean Mine Experiment

To demonstrate the real world performance and application of our approach, an experiment was conducted in a subterranean mine environment in conditions of darkness and heavy airborne dust. On-board illumination was utilized for visible light cameras to function in the otherwise completely dark underground mine environment. To provide a measure for ground–truth, pole markers were placed along the direction of the mine shaft with the final marker position being $50.0\textrm{m}$ away from the robot take-off position. The estimated robot trajectory during the experiment is shown in Figure 11. A final position error of $0.4857\textrm{m}$ was estimated for the robot navigation along the direction of the mine shaft.

Similar to the analysis of indoor experiments, we conducted 10 trials on the collected data during the experiment to verify the effect of our approach. The respective plot is shown in Figure 9. As shown, the median number of features inserted into the filter state during the mission length using the proposed feature selection mask is lower as compared to not using it. Similarly, an ANOVA test was again performed on the two populations resulting in a $p\text{-}value<0.00001$, validating that our approach makes a statistically significant difference on the selection of features to be used for estimation of odometry for the aerial robot.

Similarly, Figure 10 shows the evolution of the pose uncertainty, using the D-Optimality criterion, over the trajectory of the aerial robot navigating through the subterranean mine environment. It is highlighted that the application of our feature selection mask results in a slowdown of the growth of pose uncertainty.

5 Conclusions

In this paper, a solution to enable autonomous navigation of aerial robots in environments that are not only GPS–denied but also visually–degraded is presented. A multi–modal sensor fusion approach is proposed and relies on the fusion of visible light and thermal vision sensors, alongside inertial measurement cues. The algorithm utilizes information from both the visible and thermal spectra for landmark detection and prioritizes feature extraction from informative image regions based on a metric over spatial entropy. Through extensive experimental evaluation we demonstrate that the described solution enables reliable and accurate robot localization in challenging GPS–denied, possibly subterranean, dark, textureless and obscurants–filled environments including underground mines.

Acknowledgements.

This material is based upon work related to the Mine Inspection Robotics project sponsored by the Nevada Knowledge Fund administered by the Governor’s Office of Economic Development.