Superpixel Segmentation: A Long-Lasting Ill-Posed Problem

TL;DR

This paper reveals that superpixel segmentation is inherently ill-posed due to shape and size constraints, critiques current evaluation methods, and demonstrates competitive results with general models like SAM, prompting a rethink of superpixel properties.

Contribution

It identifies superpixel segmentation as an ill-posed problem, critiques existing evaluation frameworks, and shows that general models can perform competitively without specialized training.

Findings

Superpixel segmentation is fundamentally ill-posed due to shape and size constraints.

Current evaluation metrics often misrepresent superpixel quality.

General models like SAM can achieve competitive superpixel results without dedicated training.

Abstract

For many years, image over-segmentation into superpixels has been essential to computer vision pipelines, by creating homogeneous and identifiable regions of similar sizes. Such constrained segmentation problem would require a clear definition and specific evaluation criteria. However, the validation framework for superpixel methods, typically viewed as standard object segmentation, has rarely been thoroughly studied. In this work, we first take a step back to show that superpixel segmentation is fundamentally an ill-posed problem, due to the implicit regularity constraint on the shape and size of superpixels. We also demonstrate through a novel comprehensive study that the literature suffers from only evaluating certain aspects, sometimes incorrectly and with inappropriate metrics. Concurrently, recent deep learning-based superpixel methods mainly focus on the object segmentation task at the expense of regularity. In this ill-posed context, we show that we can achieve competitive results using a recent architecture like the Segment Anything Model (SAM), without dedicated training for the superpixel segmentation task. This leads to rethinking superpixel segmentation and the necessary properties depending on the targeted downstream task.