What Holds Back Open-Vocabulary Segmentation?

TL;DR

This paper investigates the limitations of open-vocabulary segmentation models, identifies key bottlenecks hindering their progress, and proposes oracle components to analyze and address these challenges, providing insights for future research.

Contribution

It introduces novel oracle components that leverage groundtruth data to identify and decouple bottlenecks in open-vocabulary segmentation models.

Findings

Performance plateau explained by identified bottlenecks

Oracle components reveal key failure points

Insights suggest promising directions for future improvements

Abstract

Standard segmentation setups are unable to deliver models that can recognize concepts outside the training taxonomy. Open-vocabulary approaches promise to close this gap through language-image pretraining on billions of image-caption pairs. Unfortunately, we observe that the promise is not delivered due to several bottlenecks that have caused the performance to plateau for almost two years. This paper proposes novel oracle components that identify and decouple these bottlenecks by taking advantage of the groundtruth information. The presented validation experiments deliver important empirical findings that provide a deeper insight into the failures of open-vocabulary models and suggest prominent approaches to unlock the future research.