Exploring Structural Degradation in Dense Representations for Self-supervised Learning

TL;DR

This paper investigates the phenomenon where longer training in self-supervised learning can degrade dense prediction performance, introduces a new metric for evaluation, and proposes methods to mitigate this issue, improving model selection and regularization.

Contribution

It identifies the Self-supervised Dense Degradation phenomenon, proposes the Dense representation Structure Estimator (DSE) for better evaluation, and develops DSE-based strategies to enhance dense task performance.

Findings

Model selection with DSE improves mIoU by 3.0% on average.

DSE correlates strongly with downstream dense task performance.

DSE regularization reduces the impact of dense degradation across methods.

Abstract

In this work, we observe a counterintuitive phenomenon in self-supervised learning (SSL): longer training may impair the performance of dense prediction tasks (e.g., semantic segmentation). We refer to this phenomenon as Self-supervised Dense Degradation (SDD) and demonstrate its consistent presence across sixteen state-of-the-art SSL methods with various losses, architectures, and datasets. When the model performs suboptimally on dense tasks at the end of training, measuring the performance during training becomes essential. However, evaluating dense performance effectively without annotations remains an open challenge. To tackle this issue, we introduce a Dense representation Structure Estimator (DSE), composed of a class-relevance measure and an effective dimensionality measure. The proposed DSE is both theoretically grounded and empirically validated to be closely correlated with the downstream performance. Based on this metric, we introduce a straightforward yet effective model selection strategy and a DSE-based regularization method. Experiments on sixteen SSL methods across four benchmarks confirm that model selection improves mIoU by $3.0\%$ on average with negligible computational cost. Additionally, DSE regularization consistently mitigates the effects of dense degradation. Code is available at https://github.com/EldercatSAM/SSL-Degradation.