Inference-Time Temporal Probability Smoothing for Stable Video Segmentation with SAM2 under Weak Prompts

TL;DR

This paper introduces a lightweight, inference-time method to enhance temporal stability in SAM2-based video segmentation under weak prompts by using optical flow and uncertainty estimates, without retraining.

Contribution

It proposes a novel, model-agnostic temporal probability smoothing technique that improves stability of video segmentation predictions during inference.

Findings

Significant improvement in temporal stability metrics across diverse videos.

Maintains spatial accuracy while enhancing temporal coherence.

Method is suitable for real-time interactive applications.

Abstract

Interactive video segmentation models such as SAM2 have demonstrated strong generalization across diverse visual domains. However, under weak user supervision, for example, when sparse point prompts are provided on a single frame, their predictions often suffer from temporal instability, including flickering boundaries, object dropout, and inconsistent object extents across frames. These issues limit their reliability in downstream video understanding and control applications. In this paper, we propose an inference-time temporal probability smoothing method that improves the temporal stability of SAM2-based video segmentation without retraining or architectural modification. Our approach operates directly on per-frame segmentation probability maps and leverages optical-flow-based motion warping together with pixel-wise uncertainty estimates derived from segmentation entropy, and forward-backwards flow consistency. These signals are used to adaptively blend current-frame predictions with motion-aligned historical estimates, yielding temporally coherent segmentation outputs under weak prompts. We evaluate the proposed method on four diverse video sequences using a comprehensive set of frame-wise and temporal stability metrics, including motion-compensated IoU, boundary consistency, object persistence, and area volatility. Experimental results demonstrate consistent improvements in temporal stability over vanilla SAM2 inference while preserving spatial accuracy. The proposed framework is lightweight, model-agnostic, and well-suited for real-time, interactive video segmentation.