# Aerial image segmentation of embankment dams based on multispectral remote sensing: a case study in the Belo Monte Hydroelectric Complex, Pará, Brazil

## TL;DR

This paper presents a method for automatically analyzing land cover on earth-rock dams using aerial multispectral imaging and machine learning, tested at a Brazilian hydroelectric complex.

## Contribution

A novel framework for land cover segmentation of embankment dams using UAV-based multispectral data and random forest models.

## Key findings

- The random forest model achieved 90.9% mean IoU for binary segmentation and 91.1% for multiclass segmentation.
- Post-processing improved mean IoU to 93.2% for binary and 91.9% for multiclass segmentation.
- Non-visible band reflectance data significantly contributed to model performance.

## Abstract

Visual inspection is essential to ensure the stability of earth-rock dams. Periodic visual assessment of this type of structure through vegetation cover analysis is an effective monitoring method. Recently, multispectral remote sensing data and machine learning techniques have been applied to develop methodologies that enable automatic vegetation analysis and anomaly detection based on computer vision. As a first step toward this automation, this study introduces a methodology for land cover segmentation of earth-rock embankment dam structures within the Belo Monte Hydroelectric Complex, located in the state of Pará, northern Brazil. Random forest (RF) ensemble models were trained on manually annotated data captured by a multispectral sensor embedded in an uncrewed aerial vehicle (UAV). The main objectives of this study are to assess the classification performance of the algorithm in segmenting earth-rock dams and the contribution of non-visible band reflectance data to the overall model performance. A comprehensive feature engineering and ranking approach is presented to select the most descriptive features that represent the four dataset classes. Model performance was assessed using classical performance metrics derived from the confusion matrix, such as accuracy, Kappa coefficient, precision, recall, F1-score, and intersection over union (IoU). The final RF model achieved 90.9% mean IoU for binary segmentation and 91.1% mean IoU for multiclass segmentation. Post-processing techniques were applied to refine the predicted masks, enhancing the mean IoU to 93.2% and 91.9%, respectively. The flexible methodology presented in this work can be applied to different scenarios when treated as a framework for pixel-wise land cover classification, serving as a crucial step toward automating visual inspection processes. The implementation of automated monitoring solutions improves the visual inspection process and mitigates the catastrophic consequences resulting from dam failures.