Zero-Shot Edge Detection with SCESAME: Spectral Clustering-based Ensemble for Segment Anything Model Estimation

TL;DR

This paper introduces SCESAME, a zero-shot edge detection method leveraging the Segment Anything Model, which effectively overcomes overdetecting edges and achieves performance comparable to human and CNN-based methods.

Contribution

The paper presents a novel spectral clustering-based ensemble approach for zero-shot edge detection using SAM, addressing overdetecting edges in automatic mask generation.

Findings

Achieves almost human-level performance on BSDS500 dataset.

Performs comparably to recent CNN-based methods on NYUDv2.

Effectively enhances SAM's utility for edge detection.

Abstract

This paper proposes a novel zero-shot edge detection with SCESAME, which stands for Spectral Clustering-based Ensemble for Segment Anything Model Estimation, based on the recently proposed Segment Anything Model (SAM). SAM is a foundation model for segmentation tasks, and one of the interesting applications of SAM is Automatic Mask Generation (AMG), which generates zero-shot segmentation masks of an entire image. AMG can be applied to edge detection, but suffers from the problem of overdetecting edges. Edge detection with SCESAME overcomes this problem by three steps: (1) eliminating small generated masks, (2) combining masks by spectral clustering, taking into account mask positions and overlaps, and (3) removing artifacts after edge detection. We performed edge detection experiments on two datasets, BSDS500 and NYUDv2. Although our zero-shot approach is simple, the experimental results on BSDS500 showed almost identical performance to human performance and CNN-based methods from seven years ago. In the NYUDv2 experiments, it performed almost as well as recent CNN-based methods. These results indicate that our method effectively enhances the utility of SAM and can be a new direction in zero-shot edge detection methods.