HRSAM: Efficient Interactive Segmentation in High-Resolution Images

TL;DR

HRSAM is a lightweight, high-resolution interactive segmentation model that generalizes well across resolutions, offering faster processing and improved accuracy over existing models through innovative attention and extrapolation techniques.

Contribution

We introduce HRSAM, a novel high-resolution segmentation model with a flexible attention framework and extrapolation capabilities, outperforming prior methods in speed and accuracy.

Findings

HRSAM surpasses previous state-of-the-art with 38% of the latency.

Extrapolation enables high-resolution generalization from low-resolution training.

Flash Swin attention accelerates processing by over 35%.

Abstract

The Segment Anything Model (SAM) has advanced interactive segmentation but is limited by the high computational cost on high-resolution images. This requires downsampling to meet GPU constraints, sacrificing the fine-grained details needed for high-precision interactive segmentation. To address SAM's limitations, we focus on visual length extrapolation and propose a lightweight model named HRSAM. The extrapolation enables HRSAM trained on low resolutions to generalize to high resolutions. We begin by finding the link between the extrapolation and attention scores, which leads us to base HRSAM on Swin attention. We then introduce the Flexible Local Attention (FLA) framework, using CUDA-optimized Efficient Memory Attention to accelerate HRSAM. Within FLA, we implement Flash Swin attention, achieving over a 35% speedup compared to traditional Swin attention, and propose a KV-only padding mechanism to enhance extrapolation. We also develop the Cycle-scan module that uses State Space models to efficiently expand HRSAM's receptive field. We further develop the HRSAM++ within FLA by adding an anchor map, providing multi-scale data augmentation for the extrapolation and a larger receptive field at slight computational cost. Experiments show that, under standard training, HRSAMs surpass the previous SOTA with only 38% of the latency. With SAM-distillation, the extrapolation enables HRSAMs to outperform the teacher model at lower latency. Further finetuning achieves performance significantly exceeding the previous SOTA.