How does Labeling Error Impact Contrastive Learning? A Perspective from Data Dimensionality Reduction

TL;DR

This paper examines how labeling errors affect contrastive learning's effectiveness, revealing negative impacts, proposing SVD-based mitigation, and providing practical augmentation strategies to balance data reduction and connectivity.

Contribution

It offers a theoretical and empirical analysis of labeling error impacts and introduces data dimensionality reduction techniques to mitigate these effects in contrastive learning.

Findings

Labeling errors significantly increase downstream classification risk.

SVD can reduce false positives but may also impair data connectivity.

Moderate embedding dimensions and data augmentation improve performance.

Abstract

In recent years, contrastive learning has achieved state-of-the-art performance in the territory of self-supervised representation learning. Many previous works have attempted to provide the theoretical understanding underlying the success of contrastive learning. Almost all of them rely on a default assumption, i.e., the label consistency assumption, which may not hold in practice (the probability of failure is called labeling error) due to the strength and randomness of common augmentation strategies, such as random resized crop (RRC). This paper investigates the theoretical impact of labeling error on the downstream classification performance of contrastive learning. We first reveal several significant negative impacts of labeling error on downstream classification risk. To mitigate these impacts, data dimensionality reduction method (e.g., singular value decomposition, SVD) is applied on original data to reduce false positive samples, and establish both theoretical and empirical evaluations. Moreover, it is also found that SVD acts as a double-edged sword, which may lead to the deterioration of downstream classification accuracy due to the reduced connectivity of the augmentation graph. Based on the above observations, we give the augmentation suggestion that we should use some moderate embedding dimension (such as $512, 1024$ in our experiments), data inflation, weak augmentation, and SVD to ensure large graph connectivity and small labeling error to improve model performance.