Prompt-Based Segmentation at Multiple Resolutions and Lighting Conditions using Segment Anything Model 2

TL;DR

This study evaluates the effectiveness of prompt-based segmentation models, especially SAM 2 and SAM 2.1, for solar panel detection in aerial imagery under various lighting and resolution conditions, highlighting their strengths and limitations.

Contribution

It provides a comprehensive comparison of SAM 2 and SAM 2.1 with CNN and Eff-UNet for remote sensing segmentation, emphasizing zero-shot and prompt-based strategies across diverse conditions.

Findings

SAM 2.1 shows notable improvements over SAM 2, especially in challenging conditions.

User-defined prompts significantly enhance segmentation performance in low resolution data.

SAM models outperform CNN, but Eff-UNet is more resource-efficient for high-resolution automatic segmentation.

Abstract

This paper provides insights on the effectiveness of the zero shot, prompt-based Segment Anything Model (SAM) and its updated versions, SAM 2 and SAM 2.1, along with the non-promptable conventional neural network (CNN), for segmenting solar panels in RGB aerial remote sensing imagery. The study evaluates these models across diverse lighting conditions, spatial resolutions, and prompt strategies. SAM 2 showed slight improvements over SAM, while SAM 2.1 demonstrated notable improvements, particularly in sub-optimal lighting and low resolution conditions. SAM models, when prompted by user-defined boxes, outperformed CNN in all scenarios; in particular, user-box prompts were found crucial for achieving reasonable performance in low resolution data. Additionally, under high resolution, YOLOv9 automatic prompting outperformed user-points prompting by providing reliable prompts to SAM. Under low resolution, SAM 2.1 prompted by user points showed similar performance to SAM 2.1 prompted by YOLOv9, highlighting its zero shot improvements with a single click. In high resolution with optimal lighting imagery, Eff-UNet outperformed SAMs prompted by YOLOv9, while under sub-optimal lighting conditions, Eff-UNet, and SAM 2.1 prompted by YOLOv9, had similar performance. However, SAM is more resource-intensive, and despite improved inference time of SAM 2.1, Eff-UNet is more suitable for automatic segmentation in high resolution data. This research details strengths and limitations of each model and outlines the robustness of user-prompted image segmentation models.