RADSeg: Unleashing Parameter and Compute Efficient Zero-Shot Open-Vocabulary Segmentation Using Agglomerative Models

TL;DR

RADSeg leverages an agglomerative vision foundation model, RADIO, to achieve efficient zero-shot open-vocabulary segmentation with significant improvements in accuracy, speed, and parameter efficiency over prior methods.

Contribution

This work introduces RADSeg, the first comprehensive study of RADIO for zero-shot OVSS, enhancing performance through novel attention and mask refinement techniques.

Findings

RADSeg improves mIoU by 6-30% over baseline models.

RADSeg is 3.95x faster and uses 2.5x fewer parameters.

RADSeg-base outperforms larger models in accuracy with lower resource use.

Abstract

Open-vocabulary semantic segmentation (OVSS) underpins many vision and robotics tasks that require generalizable semantic understanding. Existing approaches either rely on limited segmentation training data, which hinders generalization, or apply zero-shot heuristics to vision-language models (e.g CLIP), while the most competitive approaches combine multiple models to improve performance at the cost of high computational and memory demands. In this work, we leverage an overlooked agglomerative vision foundation model, RADIO, to improve zero-shot OVSS along three key axes simultaneously: mIoU, latency, and parameter efficiency. We present the first comprehensive study of RADIO for zero-shot OVSS and enhance its performance through self-correlating recursive attention, self-correlating global aggregation, and computationally efficient RADIO SAM mask refinement. Our approach, RADSeg, achieves 6-30% mIoU improvement in the base ViT class while being 3.95x faster and using 2.5x fewer parameters. Surprisingly, RADSeg-base (106M) outperforms previous combinations of huge vision models (850-1350M) in mIoU, achieving state-of-the-art accuracy with substantially lower computational and memory cost.