Bringing SAM to new heights: Leveraging elevation data for tree crown segmentation from drone imagery

TL;DR

This paper explores enhancing tree crown segmentation from drone imagery by leveraging the Segment Anything Model (SAM) and elevation data, demonstrating that integrating Digital Surface Models improves accuracy especially in plantation forests.

Contribution

It introduces BalSAM, a novel approach combining SAM with elevation data, and provides a comprehensive comparison of SAM-based methods against traditional models across diverse forest types.

Findings

SAM alone does not outperform Mask R-CNN without tuning.

Integrating DSM data improves segmentation accuracy.

End-to-end tuning of SAM shows promising results.

Abstract

Information on trees at the individual level is crucial for monitoring forest ecosystems and planning forest management. Current monitoring methods involve ground measurements, requiring extensive cost, time and labor. Advances in drone remote sensing and computer vision offer great potential for mapping individual trees from aerial imagery at broad-scale. Large pre-trained vision models, such as the Segment Anything Model (SAM), represent a particularly compelling choice given limited labeled data. In this work, we compare methods leveraging SAM for the task of automatic tree crown instance segmentation in high resolution drone imagery in three use cases: 1) boreal plantations, 2) temperate forests and 3) tropical forests. We also study the integration of elevation data into models, in the form of Digital Surface Model (DSM) information, which can readily be obtained at no additional cost from RGB drone imagery. We present BalSAM, a model leveraging SAM and DSM information, which shows potential over other methods, particularly in the context of plantations. We find that methods using SAM out-of-the-box do not outperform a custom Mask R-CNN, even with well-designed prompts. However, efficiently tuning SAM end-to-end and integrating DSM information are both promising avenues for tree crown instance segmentation models.