SketchYourSeg: Mask-Free Subjective Image Segmentation via Freehand Sketches

TL;DR

SketchYourSeg introduces a mask-free, sketch-based image segmentation framework that leverages freehand sketches for precise, user-guided segmentation across image galleries, addressing annotation and spatial specificity challenges.

Contribution

It pioneers a sketch-based, mask-free segmentation method that integrates SBIR and foundation models for multi-granular, user-guided image segmentation.

Findings

Outperforms existing segmentation methods on multiple benchmarks.

Eliminates the need for pixel-perfect annotations during training.

Enables precise, user-guided segmentation using freehand sketches.

Abstract

We introduce SketchYourSeg, a novel framework that establishes freehand sketches as a powerful query modality for subjective image segmentation across entire galleries through a single exemplar sketch. Unlike text prompts that struggle with spatial specificity or interactive methods confined to single-image operations, sketches naturally combine semantic intent with structural precision. This unique dual encoding enables precise visual disambiguation for segmentation tasks where text descriptions would be cumbersome or ambiguous -- such as distinguishing between visually similar instances, specifying exact part boundaries, or indicating spatial relationships in composed concepts. Our approach addresses three fundamental challenges: (i) eliminating the need for pixel-perfect annotation masks during training with a mask-free framework; (ii) creating a synergistic relationship between sketch-based image retrieval (SBIR) models and foundation models (CLIP/DINOv2) where the former provides training signals while the latter generates masks; and (iii) enabling multi-granular segmentation capabilities through purpose-made sketch augmentation strategies. Our extensive evaluations demonstrate superior performance over existing approaches across diverse benchmarks, establishing a new paradigm for user-guided image segmentation that balances precision with efficiency.