Probing the Mid-level Vision Capabilities of Self-Supervised Learning

TL;DR

This paper systematically evaluates the mid-level vision capabilities of 22 self-supervised learning models across 8 tasks, revealing insights into their strengths, weaknesses, and factors influencing performance, which guides future SSL research.

Contribution

Introduces benchmark protocols and conducts a comprehensive evaluation of SSL models on mid-level vision tasks, highlighting the gap with high-level task performance and factors affecting capabilities.

Findings

Weak correlation between mid-level and high-level task performance

Some SSL models excel in both mid-level and high-level tasks

Pretraining objectives and architectures significantly influence mid-level vision capabilities

Abstract

Mid-level vision capabilities - such as generic object localization and 3D geometric understanding - are not only fundamental to human vision but are also crucial for many real-world applications of computer vision. These abilities emerge with minimal supervision during the early stages of human visual development. Despite their significance, current self-supervised learning (SSL) approaches are primarily designed and evaluated for high-level recognition tasks, leaving their mid-level vision capabilities largely unexamined. In this study, we introduce a suite of benchmark protocols to systematically assess mid-level vision capabilities and present a comprehensive, controlled evaluation of 22 prominent SSL models across 8 mid-level vision tasks. Our experiments reveal a weak correlation between mid-level and high-level task performance. We also identify several SSL methods with highly imbalanced performance across mid-level and high-level capabilities, as well as some that excel in both. Additionally, we investigate key factors contributing to mid-level vision performance, such as pretraining objectives and network architectures. Our study provides a holistic and timely view of what SSL models have learned, complementing existing research that primarily focuses on high-level vision tasks. We hope our findings guide future SSL research to benchmark models not only on high-level vision tasks but on mid-level as well.