AoP-SAM: Automation of Prompts for Efficient Segmentation

TL;DR

AoP-SAM automates prompt generation for the Segment Anything Model, significantly improving efficiency and accuracy in image segmentation without manual prompts or fine-tuning.

Contribution

This paper introduces AoP-SAM, a novel automatic prompt generation method that enhances SAM's usability and efficiency for real-world segmentation tasks.

Findings

Substantial improvement in prompt generation efficiency

Enhanced mask accuracy across multiple datasets

Reduced computational overhead in segmentation process

Abstract

The Segment Anything Model (SAM) is a powerful foundation model for image segmentation, showing robust zero-shot generalization through prompt engineering. However, relying on manual prompts is impractical for real-world applications, particularly in scenarios where rapid prompt provision and resource efficiency are crucial. In this paper, we propose the Automation of Prompts for SAM (AoP-SAM), a novel approach that learns to generate essential prompts in optimal locations automatically. AoP-SAM enhances SAM's efficiency and usability by eliminating manual input, making it better suited for real-world tasks. Our approach employs a lightweight yet efficient Prompt Predictor model that detects key entities across images and identifies the optimal regions for placing prompt candidates. This method leverages SAM's image embeddings, preserving its zero-shot generalization capabilities without requiring fine-tuning. Additionally, we introduce a test-time instance-level Adaptive Sampling and Filtering mechanism that generates prompts in a coarse-to-fine manner. This notably enhances both prompt and mask generation efficiency by reducing computational overhead and minimizing redundant mask refinements. Evaluations of three datasets demonstrate that AoP-SAM substantially improves both prompt generation efficiency and mask generation accuracy, making SAM more effective for automated segmentation tasks.