Detecting Invasive Insects with Unmanned Aerial Vehicles

TL;DR

This paper introduces an automated UAV-based system employing UV lighting and computer vision to detect invasive insects, significantly improving detection speed and accuracy over traditional manual methods, thereby enhancing understanding of insect migration.

Contribution

The paper presents a novel UAV system integrating UV lighting and lightweight vision algorithms for automated insect detection, surpassing existing manual techniques in efficiency and precision.

Findings

High detection precision and recall in field conditions

Faster and more accurate than manual search methods

Enables comprehensive study of insect migration patterns

Abstract

A key aspect to controlling and reducing the effects invasive insect species have on agriculture is to obtain knowledge about the migration patterns of these species. Current state-of-the-art methods of studying these migration patterns involve a mark-release-recapture technique, in which insects are released after being marked and researchers attempt to recapture them later. However, this approach involves a human researcher manually searching for these insects in large fields and results in very low recapture rates. In this paper, we propose an automated system for detecting released insects using an unmanned aerial vehicle. This system utilizes ultraviolet lighting technology, digital cameras, and lightweight computer vision algorithms to more quickly and accurately detect insects compared to the current state of the art. The efficiency and accuracy that this system provides will allow for a more comprehensive understanding of invasive insect species migration patterns. Our experimental results demonstrate that our system can detect real target insects in field conditions with high precision and recall rates.

Equations

$$\mathbb{V}=\left\{{\mathbb{I}}_1,{\mathbb{I}}_2,...,{\mathbb{I}}_n\right\},$$

$${\mathbb{I}}_i^{(s)}={\mathbb{I}}_i^{(d)}\odot {\mathbb{I}}_i^{(w)}$$

$$t_m=\underset{{\mathbb{I}}_i\in \mathbb{V}}{\max }{t}_{o,i},$$

Full text

I Introduction

Invasive species are often inadvertently transported from their native environments to new habitats where they tend to have severe negative impacts on food security, public health, economic interests, and native species biodiversity [13]. As an example, in the United States, annual economic losses from invasive species are estimated at $120 billion [26], and these severe impacts are predicted to continue [23].

Understanding invasive pest migration patterns is a key element to the mitigation of their damage to both natural environments and agricultural production. Insect species are highly variable in their dispersal capacity. For example, invasive brown marmorated stink bugs (BMSB) are extremely mobile and can fly 115km in 24 hours [12, 35], whereas invasive emerald ash borers typically fly less than 100m in natural habitats [16]. Therefore, the dispersal capacity of newly identified invasive insect species must be well-defined to determine what strategy (eradication or management) to initiate.

Research in this area has focused on the influence of wind and temperature in long-range migrations, [33, 5], neglecting the study of short dispersal patterns. Although there are studies about insect dispersal, they are limited to mark-release-recapture techniques – a laborious, time-consuming, and error-prone task. In addition, because of the labor-intensive nature of the process, very few samples of the marked insects can be recaptured, typically < 5%. This extremely low recapture rate negatively impacts the accuracy of the resulting dispersal models [29, 17, 20].

In this paper, we describe a novel aerial system that attempts to solve the limitations of the manual methods described above. The proposed approach uses an Unmanned Aerial Vehicle (UAV) to scan a region of interest and detect target insects using a recently developed system of ultraviolet (UV) lights. With the UAV equipped with a UV light source and video camera, we record an aerial video of the area. We later process the video offline to obtain an accurate count of the marked insects released in the field.

The video processing pipeline described in this paper employs multi-level color channel thresholding to segment the marked insects from the background. It consists of three main steps. First, we extract the region of interest (ROI) from the video frames, keeping only the region surrounding the area illuminated by the UV light. Then, we identify the illuminated insects using color channel thresholding in the RGB and HSV color spaces. Finally, we use watershed segmentation [32] to separate nearby insects that might have been identified as a single insect in the previous step. Our experiments demonstrate the detection of marked insects in field conditions.

The contributions and impact of this work are as follows: 1) We propose a novel data acquisition system based on a UAV that allows us to quickly search for insects in a large area of interest, 2) we design a low complexity computer vision algorithm to robustly detect and count the insects observed by our system, and 3) we evaluate the data acquisition system in open field situations using industry standard benchmarks. The impact of this work represents a considerably faster, economical, and accurate alternative to current manual mark-release-recapture methods for insect pest monitoring. Moreover, as there is no need to recapture the insects after they are detected, our non-intrusive method allows for researchers to dynamically localize insects over time.

II Related Work

For over a century, entomologists have used mark-release-recapture techniques to track insect movement. These initial efforts traditionally used paint and dyes as markers [10]. In recent years, protein markers were developed to quickly mark thousands of individual insects within minutes [8, 7], enabling more sophisticated studies. However, protein marking involves inherent challenges. First, the proteins degrade quickly under field conditions. In addition, a time-consuming process based on enzyme-linked immunosorbent assays (ELISA) is needed to determine whether a specific protein marker is present [4]. Finally, because of the ELISA test itself, the target insect is ultimately destroyed thereby eliminating the possibility of tracking the same insect more than once.

II-A Ultraviolet light tracking

The use of methods based on fluorescent pigment is a simple alternative that can effectively mark millions of insects in a short amount of time [31, 9]. Fluorescent markings (see Figure 1) glow under UV light by transforming radiant energy from the UV band to longer wavelengths that are detectable by the human eye. However, detection distances of fluorescent powders with UV light sources are significantly short – less than a few meters – even in dark conditions. Thus, many studies employing fluorescent pigment marking require recaptured organisms to be taken to the laboratory to verify marks under UV lamps [14, 19], rather than in the insect’s initial location.

Rice et al. described in [28] a novel method for detecting fluorescent-marked insects. They use hand-held UV lasers with focusable lenses which allow the beam to be widened significantly for a larger search area and increased detection distance up to 40m. The method also enables non-destructive scanning from the ground of previously inaccessible habitats, including tree canopies and aquatic habitats. These contributions have resulted in increased recovery rates and the amount of information gathered. Additionally, utilizing UAVs in conjunction with UV lights has been proposed in other research applications. One such example is using a UAV to inspect transmission lines, while implementing a UV light to highlight problems with high voltage coronas [30]. This demonstrates the advantage of using UV light to assist with UAV vision operations.

II-B Agricultural applications of unmanned aerial vehicles

Recently, there has been increasing interest in the use of UAVs in agricultural research, with predictions that it will revolutionize spatial ecology [2]. UAVs provide the ability to collect remote data at an unprecedented scale and sampling rates at a fraction of the cost of previous methods such as satellites or manned aircraft [36]. Current examples of research applications include methods of classification and monitoring, such as weed classification [15] and applying a multispectral imaging system for crop monitoring [18]. Similar to our proposed system of mapping insects in a field, other related research topics include aerial mapping of rice crops [22]. Similar agricultural mapping techniques involve using swarms of UAVs to map weeds in large fields [1] and terrain surveying of disjointed fields aided by UAVs and path planning [34]. Lastly, UAVs can provide a system for detection and estimation, such as using the vehicles to provide vital data on plant stress [3].

The UAV sensing modality is determined by the application needs as well as the payload capacity of the aircraft. Hyperspectral and multispectral cameras are among the most popular sensors, but regular and thermal cameras, lidar and radar imaging and even chemical sensing have been applied in specialized areas [24]. Recently, radio tags have been employed to track invasive fish species using UAVs [27]. However, no applications of UAVs to track insect movement and dispersal in field conditions have been proposed so far.

III Insect detection system

In this section, we describe the hardware and software components of the proposed detection system. We designed the system with the assumption that insects have been previously coated with a fluorescent pigment. It is also assumed the data acquisition process is carried out at night with limited artificial illumination sources other than the UV light system. The insect detection process consists of two primary components, the aerial data acquisition and the insect detection and segmentation algorithm. Data acquisition consists of flying the UAV over the area under study with the camera and UV light source facing down, illuminating and filming any coated insects on the ground. Once data acquisition is completed, the video files are downloaded onto a workstation and used as input to the software pipeline. This pipeline utilizes multi-channel thresholding techniques to detect all the insects in the video.

III-A Hardware components

Our system consists of a UAV equipped with a set of ten ultraviolet lights and a high-resolution camera and gimbal unit. The UAV is a Matrice 100 model – a four-propeller UAV that is ready-to-fly off the shelf yet still fully customizable (see Figure 2). A benefit of the Matrice 100 is its extended flight time of up to 40 minutes. This is essential for the plausibility of our proposed system, as large fields will take considerable time for the UAV to cover.

The camera and gimbal unit is the Zenmuse X3. The Zenmuse X3 is an all-in-one gimbal and camera tool that can produce videos with $4096\times 2160$ pixel resolution and 3 axis movement stability. The gimbal is set to be pointed perpendicular to the ground at all times. The 3-axis stability allows for clear, non-blurry footage to be collected while the UAV is moving. The camera’s parameters can be adjusted through the DJI Ground Station Pro app for iOS devices. All conducted tests would be in a low-light environment, so we used an ISO rating of 1600 and a shutter speed of 1/25 second for all video footage.

The UV illumination system is attached to the bottom frame of the UAV and can be controlled by a remote transmitter. The UV lights consist of high-power violet LEDs with a wavelength of 395 nm. Each LED is encapsulated by a narrowing lens that focuses the light emitted by the LED. The lights are attached to a $12.7\times 30.5$ cm aluminum heatsink (see Figure 2) and a set of four 3.7V Li-ion batteries with 3400mAh capacity to supply power to the UV lights. The power source is controlled with an RF remote control relay switch fastened between the heatsink and battery pack. The light system is secured to the UAV using four fastening screws so that the UV lights are pointed downward.

Although the camera is mounted on a gimbal, the illumination system is not. This inevitably causes the UV projection to always be perpendicular to the UAV rather than the ground when the UAV is in motion. To minimize the movement of the UV-illuminated area, we installed the UV LED array in the center of the vehicle’s body frame, which corresponds to the pivot point of the pitch, roll, and yaw motions. This layout also improves flight dynamics and stability, as it reduces the moment of inertia caused by the weight of the system (1029g with batteries). Additionally, the UAV is flown at the relatively slow speed of 1 m/s to reduce the angular tilt caused by the forward movement of the UAV.

III-B Software pipeline

The video recorded during the flight, $\mathbb{V}$, consists of a sequence of $n$ frames

[TABLE]

where each frame $\mathbb{I}_{i}\in\mathbb{R}^{h\times w\times c}$, with $h$ and $w$ representing the height and width of the image, and $c$ its number of color channels. Thus, each image pixel $p_{x,y}^{i}$ is a $c$-dimensional vector, where $x$ and $y$ are its coordinates in the image. The output video $\mathbb{V}$ is then used as the input to Algorithm 1, which summarizes the processing steps carried out after each data collection flight.

1) Step 1: Region of interest selection: To reduce the execution time of our software pipeline, a region of interest is selected in relation to the middle of each frame. A supplemental advantage of selecting a region of interest is reducing the amount of noise on the edges of the frame since all the insects should be visible only within the ultraviolet light-illuminated portion of the frame. The field of view (FOV) of the light system ($30{{}^{\circ}}$) is significantly narrower than the FOV of the Zenmuse X3 ($94{{}^{\circ}}$). Therefore, we determined the size of the image $\mathbb{I}_{i}^{(r)}$, $h^{\prime}\times w^{\prime}$, experimentally to ensure the UV light projection always remains within the region of interest. The determined values were $720$ x 720 pixels, which corresponds to 25% of the original image size. This allows for a relative angle of up to 16° between the camera and the light beam.

2) Step 2: Thresholding: To identify the insects in $\mathbb{V}$, we segment the pixels of $\mathbb{I}_{i}^{(r)}$ as belonging to the foreground (insects) or background using simple color thresholding. This is plausible because of the distinctive color of the marked insects with respect to the background. The result of this step is $\mathbb{I}_{i}^{(d)}$, an image displaying the detected insects over a black background, as shown in Figure 3(a). To achieve this result, we used two different color space representations, RGB (red, green, blue) and HSV (hue, saturation, value).

The fluorescent powder used shows a pink color when illuminated with UV light. Pink exhibits very low green values compared with its red and blue components. We enforce this relation between the three color channels in our insect thresholding algorithm. That is, let $p=\left[p_{r},p_{g},p_{b}\right]$ be a pixel in $\mathbb{I}_{i}^{(r)}$, then $p$ is considered a foreground pixel as long as $p_{r}>p_{b}>p_{g}$.

RGB thresholding alone, however, is not robust enough against reflective surfaces such as dewy grass. In our case, the reflection of the UV light (violet color) on some surfaces can result in dark pink pixels, which can be mistaken for insects. To improve the robustness of our method, we incorporate HSV color space thresholding. We use a set of nine calibration images of insects illuminated by the UV light on a non-reflective background to determine $\mu_{v}$, the average values of the brightness component of the reflected light. We then use this value as a lower detection threshold. That is, let $p=\left[p_{h},p_{s},p_{v}\right]$ be a pixel in $\mathbb{I}_{i}^{(h)}$, the HSV representation of $\mathbb{I}_{i}^{(r)}$, then for $p$ to be considered a foreground pixel it has to satisfy $p_{v}>\mu_{v}$. The determined value $\mu_{v}$ was 40. Additionally, the value channel threshold allows for further differentiation from the black background, as black pixels have very low brightness values. We have determined experimentally that our method is not sensitive to the number of images used for calibration. The method shows identical results if the number of calibration images is between five and eighteen.

The bright red and pink color of the marked insects correspond to both very high and very low values in the hue channel, leading us to choose an upper and lower threshold requirement for hue levels. The values are denoted by $h_{ut}$ and $h_{lt}$ for the upper and lower hue thresholds, respectively. Inversely, we found that marked insects typically exhibited saturation values in the middle of the saturation spectrum. Thus, we again use upper and lower thresholds for the saturation channel, denoted by $s_{ut}$ and $s_{lt}$, respectively. That is, for a pixel $p$ to be considered a foreground pixel, it must satisfy $p_{h}>h_{ut},$ $p_{h}<h_{lt}$, $s_{lt}<p_{s}<s_{ut}$. The following optimal threshold values were found throughout our experiments: $h_{ut}$ = 220, $h_{lt}$ = 25, $s_{lt}$ = 90, $s_{ut}$ = 255.

3) Step 3: Segmentation: Thresholding alone is not sufficient to identify multiple insects in close proximity to each other. Groups of coated insects represent very brightly colored areas, and the gap between them is often detected as part of the foreground (Figure 3 (a)). To overcome this difficulty, we use watershed segmentation [25] to split groups of insects into individual detections. By applying watershed segmentation to $\mathbb{I}_{i}^{(r)}$, we generate an image of local intensity peaks, $\mathbb{I}_{i}^{(w)}$ (Figure 3 (b)). Then we use $\mathbb{I}_{i}^{(d)}$ as a mask for $\mathbb{I}_{i}^{(w)}$ so that only the peaks that are inside the thresholded blobs of $\mathbb{I}_{i}^{(d)}$ are maintained. The new image $\mathbb{I}_{i}^{(s)}$ is given by

[TABLE]

where $\odot$ represents elementwise multiplication. As Figure 3 (c) illustrates, all detected insects can be clearly identified in the resulting image $\mathbb{I}_{i}^{(s)}$.

4) Step 4: Counting: To count the detected insects $d_{i}$ in $\mathbb{I}_{i}^{(s)}$, we first binarize the image using the thresholds and segmentation steps from steps two and three. We subsequently perform blob analysis using MATLAB’s Computer Vision Toolbox111https://www.mathworks.com/products/computer-vision.html, thereby counting the number of blobs (groups of foreground pixels) in the image. This step additionally allows us to apply a detection size filter to discard blobs that contain fewer than a certain number of pixels $n_{p}$.

To calculate the total number of insects in the video, we create a global counter that is incremented whenever the number of blobs in $\mathbb{I}_{i}^{(s)}$ is higher than the number of blobs in $\mathbb{I}_{i-1}^{(s)}$. Figure 3 (d) shows the result of this step. Note that this method does not take into consideration potential double-counting caused by scanning an image area multiple times. Addressing this limitation is part of our future work.

Finally, for visualization purposes, we highlight each blob with a green bounding box, as show in Figure 3 (e).

IV Experimental Results

In this section, we evaluate the performance of the proposed system in our testing location, a grassy outdoor field. We compare our method with a baseline approach that uses Otsu’s algorithm [21] to determine the optimal luminance threshold for the video.

IV-A Baseline method

To the best of our knowledge, no automated methods to segment fluorescent insects currently exist. Therefore, we select Otsu’s algorithm as a baseline approach for comparison. Figures 4(a) and 4(b) illustrate that when no fluorescent target is illuminated, Otsu’s threshold generates significant amounts of clutter. To address this problem, we search the entire video for the maximum threshold value $t_{m}$ obtained applying Otsu’s algorithm to each frame and use that value for all the video frames on a subsequent pass, i.e.,

[TABLE]

where $t_{o,i}$ is Otsu’s threshold for frame $\mathbb{I}_{i}$. Figure 4(c) shows the results of applying this method to the two images shown in Figure 4(a).

IV-B Target species and marking procedure

All the experiments described in this section were carried out using real fluorescent-marked insects. In particular, we used 36 fluorescent-marked brown marmorated stink bugs (BMSB) with sizes between 13.5 x 7 mm to 16 x 8 mm (Figure 5(a)). To coat the insects, we individually placed them in a plastic cylinder with 2 g of red fluorescent powder (BioQuip, Rancho Dominguez, CA) and gently shook it for five seconds.

IV-C Test site

Due to regulatory restrictions and local weather conflicts, opportunities for testing the method in an outdoor field at night, are limited. Therefore, we conducted our tests on a campus field located at Marquette University in Milwaukee, Wisconsin. To mitigate the impact of the illumination at the edges of the field, all the tests were performed on a $15.24\times 15.24$ meter area at the center of the field (Figures 6(a) and 6(b)). In the test area, the average grass height was 9 cm, the average luminous value was 0.1 lux, and the average wind speed during testing was 1.4 km/h.

To make the conditions in our testing field as similar as practically possible to the natural fields insects would inhabit, we placed some insects on top of the grass gently and some closer to the base of the grass, causing them to be slightly occluded from above by the grass. The arrangement of the insects can be seen in Figure 6(a). A total of 36 insects were arranged in six rows of six insects each with gaps of 3.05 meters between each other. The data was collected with the UAV flying at a height of 10 meters. The resolution of the video was $4096\times 2160$. Under these conditions, the average width of the insects was 7.8 pixels and the average length was 14.6 pixels. The standard deviation of these values were 2.7 pixels in width and 5.0 pixels in length. The UAV flying speed was 1m/s and the video frame rate was 24 frames per second. Once the system has been set up, the entire data collection session can be performed in approximately three minutes.

IV-D Mission planning

To ensure that all flight tests are performed in a similar pattern with minimal human error, we use the DJI Ground Station (GS) Pro app222https://www.dji.com/ground-station-pro to prepare a flight pattern ahead of the experiments. The tool provides planning flexibility, allowing many parameters to be adjusted, such as flight speed, flight altitude, and corner rounding radius. In this project, we used the WayPoint Route mission type (Figure 6(b)), which provides a simple tap-to-mark system to select the locations to which the UAV should fly. Based on the user-provided locations, the application creates a set of GPS waypoints that intercept these points while also satisfying the image acquisition requirements. As Figure 6(b) shows, in our experiments, we used 10 waypoints with a spacing of 15.24 meters in the vertical (i.e., south to north) direction and 3.05 meters in the horizontal (west to east) direction.

IV-E Insect detection performance

In the evaluation of the detection performance, we chose not to rely on the GPS coordinates of the targets or the UAV as the ground truth. This approach eliminates the effects of localization and georeferencing error in our analysis. Placing insects in a formation with specific GPS coordinates is prone to error, especially with low precision methods to determine the GPS location of a specific spot in the field. Instead, we evaluate the insect detection method directly on the images by comparing the location and size of the bounding boxes around each detection $d_{i}$ in $\mathbb{I}_{i}^{(r)}$ with the ground truth boxes ($GT_{i}$). $GT_{i}$ is a $h^{\prime}\times w^{\prime}$ binary image that contains manually labeled detection targets. To be labeled as a ground truth target, an insect must be clearly visible to the naked eye within the UV light beam in the images. A detection is considered a true positive if its intersection over union (IoU) metric with respect to a ground truth bounding box is higher than 50%. We compute the average recall and precision values of the baseline and proposed methods over the entire video sequence [6]. In our experiments, the testing footage consists of 2477 frames. Figure 7 shows the the precision-recall (PR) curves for our approach and the baseline method. The area under the PR curve for our method is $0.61$ whereas for the baseline approach it is $0.36$. Our method shows a precision as high as $0.8$ for a recall rate of $0.7$, while the baseline methods has a precision of $0.28$ for the same recall value. Most of the mistakes in our method are due to the fact that an IoU of 50% is challenging to achieve for very small objects. At lower IoU levels, the results would be improved.

IV-F Computation time

The uncomplicated design of the detection algorithm allows us for quick processing of footage. The image cropping step can be executed in $O(1)$ time, and the remaining steps run in $O(s^{\prime})$ time, where $\mathit{s^{\prime}=h^{\prime}\,\mathrm{x}\,w^{\prime}}$, i.e., the resolution of the region of interest. The overall method runs in $O(s^{\prime})$ time.

V Conclusions and Future Work

We have described a novel system that combines UAVs, ultraviolet lighting systems, and computer vision algorithms to detect fluorescent-coated insects in the field. To the best of our knowledge, this is the first vision-based system that detects insects in the field. It uses an illumination system based on a UV light source to visualize the insects and a color-based detection algorithm that requires minimal calibration. The system was evaluated relative to its detection precision and recall and compared with a baseline detection approach based on Otsu’s algorithm. The proposed system corresponds to a significant advancement of the state of the art as manual insect recapture rates are much lower, even with long range laser-based systems [28].

In the future, we intend to incorporate the GPS locations of the insects as the ground truth labels and use the UAV’s GPS location and inertial measurements to generate orthographic projections of the insect locations. Being able to use GPS locations rather than labeled video footage would allow us to directly map the locations of invasive insects and monitor their dispersal.

Acknowledgment

This work is supported by Agriculture and Food Research Initiative Agricultural Engineering grant no. 2018-67021-28318 from the USDA National Institute of Food and Agriculture. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the view of the U.S. Department of Agriculture.

The authors would like to thank Weihua Liu and Scott Stewart for their assistance with data collection and annotation.