HiM2SAM: Enhancing SAM2 with Hierarchical Motion Estimation and Memory Optimization towards Long-term Tracking

TL;DR

This paper enhances the SAM2 video tracking framework with hierarchical motion estimation and memory optimization, significantly improving long-term tracking accuracy without additional training, and achieving state-of-the-art results on benchmark datasets.

Contribution

Introduces a hierarchical motion estimation strategy and memory bank optimization to improve SAM2's long-term tracking performance without extra training.

Findings

Achieves 9.6% and 7.2% relative improvements in AUC on LaSOT and LaSOText.

Demonstrates larger gains on smaller models, showing effectiveness across scales.

Provides a trainless, low-overhead method for enhanced long-term tracking.

Abstract

This paper presents enhancements to the SAM2 framework for video object tracking task, addressing challenges such as occlusions, background clutter, and target reappearance. We introduce a hierarchical motion estimation strategy, combining lightweight linear prediction with selective non-linear refinement to improve tracking accuracy without requiring additional training. In addition, we optimize the memory bank by distinguishing long-term and short-term memory frames, enabling more reliable tracking under long-term occlusions and appearance changes. Experimental results show consistent improvements across different model scales. Our method achieves state-of-the-art performance on LaSOT and LaSOText with the large model, achieving 9.6% and 7.2% relative improvements in AUC over the original SAM2, and demonstrates even larger relative gains on smaller models, highlighting the effectiveness of our trainless, low-overhead improvements for boosting long-term tracking performance. The code is available at https://github.com/LouisFinner/HiM2SAM.