FreeVPS: Repurposing Training-Free SAM2 for Generalizable Video Polyp Segmentation

TL;DR

FreeVPS introduces a training-free approach that repurposes SAM2 for stable, generalizable video polyp segmentation by mitigating error accumulation and enhancing temporal coherence in colonoscopy videos.

Contribution

It presents a novel training-free framework that combines spatial and temporal modeling for VPS, addressing error propagation and improving robustness in clinical scenarios.

Findings

Achieves state-of-the-art performance in in-domain and out-of-domain VPS tasks.

Effectively reduces false positives through intra-association filtering.

Maintains stable long-term tracking in colonoscopy videos.

Abstract

Existing video polyp segmentation (VPS) paradigms usually struggle to balance between spatiotemporal modeling and domain generalization, limiting their applicability in real clinical scenarios. To embrace this challenge, we recast the VPS task as a track-by-detect paradigm that leverages the spatial contexts captured by the image polyp segmentation (IPS) model while integrating the temporal modeling capabilities of segment anything model 2 (SAM2). However, during long-term polyp tracking in colonoscopy videos, SAM2 suffers from error accumulation, resulting in a snowball effect that compromises segmentation stability. We mitigate this issue by repurposing SAM2 as a video polyp segmenter with two training-free modules. In particular, the intra-association filtering module eliminates spatial inaccuracies originating from the detecting stage, reducing false positives. The inter-association refinement module adaptively updates the memory bank to prevent error propagation over time, enhancing temporal coherence. Both modules work synergistically to stabilize SAM2, achieving cutting-edge performance in both in-domain and out-of-domain scenarios. Furthermore, we demonstrate the robust tracking capabilities of FreeVPS in long-untrimmed colonoscopy videos, underscoring its potential reliable clinical analysis.