Fusion or Confusion? Assessing the impact of visible-thermal image fusion for automated wildlife detection

TL;DR

This study evaluates visible-thermal image fusion for automated wildlife detection, showing that fusion improves detection accuracy but faces challenges like field of view and alignment constraints, suggesting high-res visible sensors as a practical solution.

Contribution

It compares early and late fusion methods for VIS and TIR imagery in wildlife detection, demonstrating improved accuracy and discussing practical limitations and solutions.

Findings

Fusion methods improved F1 scores over VIS-only models.

Late fusion increased F1 score for nests from 90.2% to 93.0%.

Fusion achieved 90% recall in false positive identification.

Abstract

Efficient wildlife monitoring methods are necessary for biodiversity conservation and management. The combination of remote sensing, aerial imagery and deep learning offer promising opportunities to renew or improve existing survey methods. The complementary use of visible (VIS) and thermal infrared (TIR) imagery can add information compared to a single-source image and improve results in an automated detection context. However, the alignment and fusion process can be challenging, especially since visible and thermal images usually have different fields of view (FOV) and spatial resolutions. This research presents a case study on the great blue heron (Ardea herodias) to evaluate the performances of synchronous aerial VIS and TIR imagery to automatically detect individuals and nests using a YOLO11n model. Two VIS-TIR fusion methods were tested and compared: an early fusion approach and a late fusion approach, to determine if the addition of the TIR image gives any added value compared to a VIS-only model. VIS and TIR images were automatically aligned using a deep learning model. A principal component analysis fusion method was applied to VIS-TIR image pairs to form the early fusion dataset. A classification and regression tree was used to process the late fusion dataset, based on the detection from the VIS-only and TIR-only trained models. Across all classes, both late and early fusion improved the F1 score compared to the VIS-only model. For the main class, occupied nest, the late fusion improved the F1 score from 90.2 (VIS-only) to 93.0%. This model was also able to identify false positives from both sources with 90% recall. Although fusion methods seem to give better results, this approach comes with a limiting TIR FOV and alignment constraints that eliminate data. Using an aircraft-mounted very high-resolution visible sensor could be an interesting option for operationalizing surveys.