Contrastive Learning with Nasty Noise

TL;DR

This paper investigates the theoretical limits of contrastive learning when faced with adversarial noise, providing bounds on sample complexity and analyzing robustness in such challenging settings.

Contribution

It introduces a theoretical framework for understanding contrastive learning under adversarial noise, including bounds based on PAC learning and VC-dimension analysis.

Findings

Lower and upper bounds on sample complexity under adversarial noise

Data-dependent sample complexity bounds based on l2-distance

Insights into the robustness of contrastive learning algorithms

Abstract

Contrastive learning has emerged as a powerful paradigm for self-supervised representation learning. This work analyzes the theoretical limits of contrastive learning under nasty noise, where an adversary modifies or replaces training samples. Using PAC learning and VC-dimension analysis, lower and upper bounds on sample complexity in adversarial settings are established. Additionally, data-dependent sample complexity bounds based on the l2-distance function are derived.