Let Human Sketches Help: Empowering Challenging Image Segmentation Task with Freehand Sketches
Ying Zang, Runlong Cao, Jianqi Zhang, Yidong Han, Ziyue Cao, Wenjun, Hu, Didi Zhu, Lanyun Zhu, Zejian Li, Deyi Ji, Tianrun Chen

TL;DR
This paper introduces a sketch-guided interactive segmentation framework that leverages freehand sketches to improve challenging image segmentation tasks like camouflaged object detection, reducing annotation effort and enhancing performance.
Contribution
It presents a novel sketch-based annotation method, architectural modifications, and a new dataset, significantly advancing interactive segmentation and reducing annotation time.
Findings
Sketch input outperforms text and bounding boxes in segmentation accuracy.
The method reduces annotation time by up to 120 times.
The approach achieves comparable results to pixel-level annotations.
Abstract
Sketches, with their expressive potential, allow humans to convey the essence of an object through even a rough contour. For the first time, we harness this expressive potential to improve segmentation performance in challenging tasks like camouflaged object detection (COD). Our approach introduces an innovative sketch-guided interactive segmentation framework, allowing users to intuitively annotate objects with freehand sketches (drawing a rough contour of the object) instead of the traditional bounding boxes or points used in classic interactive segmentation models like SAM. We demonstrate that sketch input can significantly improve performance in existing iterative segmentation methods, outperforming text or bounding box annotations. Additionally, we introduce key modifications to network architectures and a novel sketch augmentation technique to fully harness the power of sketch…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsVisual Attention and Saliency Detection · Advanced Neural Network Applications · Augmented Reality Applications
MethodsSegment Anything Model
