Efficient Segment Anything with Depth-Aware Fusion and Limited Training Data

TL;DR

This paper introduces a lightweight RGB-D fusion approach that enhances segmentation performance using monocular depth priors, trained on significantly less data than traditional models.

Contribution

The authors develop a novel depth-aware fusion framework that improves segmentation accuracy with limited training data, reducing reliance on large-scale datasets.

Findings

Outperforms EfficientViT-SAM in accuracy

Requires only 11.2k training samples

Utilizes monocular depth priors for geometric enhancement

Abstract

Segment Anything Models (SAM) achieve impressive universal segmentation performance but require massive datasets (e.g., 11M images) and rely solely on RGB inputs. Recent efficient variants reduce computation but still depend on large-scale training. We propose a lightweight RGB-D fusion framework that augments EfficientViT-SAM with monocular depth priors. Depth maps are generated with a pretrained estimator and fused mid-level with RGB features through a dedicated depth encoder. Trained on only 11.2k samples (less than 0.1\% of SA-1B), our method achieves higher accuracy than EfficientViT-SAM, showing that depth cues provide strong geometric priors for segmentation.