Segment Anything Meets Point Tracking

TL;DR

SAM-PT introduces a novel point-centric approach for interactive video segmentation, leveraging long-term point tracking with SAM to improve zero-shot performance and interaction efficiency across multiple benchmarks.

Contribution

The paper proposes SAM-PT, a new method that uses point propagation for video segmentation, exploiting local structure information independently of object semantics.

Findings

Outperforms traditional mask propagation methods on multiple benchmarks.

Achieves better zero-shot performance in open-world video object segmentation.

Provides an efficient point-based tracking framework with publicly available code.

Abstract

The Segment Anything Model (SAM) has established itself as a powerful zero-shot image segmentation model, enabled by efficient point-centric annotation and prompt-based models. While click and brush interactions are both well explored in interactive image segmentation, the existing methods on videos focus on mask annotation and propagation. This paper presents SAM-PT, a novel method for point-centric interactive video segmentation, empowered by SAM and long-term point tracking. SAM-PT leverages robust and sparse point selection and propagation techniques for mask generation. Compared to traditional object-centric mask propagation strategies, we uniquely use point propagation to exploit local structure information agnostic to object semantics. We highlight the merits of point-based tracking through direct evaluation on the zero-shot open-world Unidentified Video Objects (UVO) benchmark. Our experiments on popular video object segmentation and multi-object segmentation tracking benchmarks, including DAVIS, YouTube-VOS, and BDD100K, suggest that a point-based segmentation tracker yields better zero-shot performance and efficient interactions. We release our code that integrates different point trackers and video segmentation benchmarks at https://github.com/SysCV/sam-pt.