Self-Supervised Learning by Curvature Alignment

TL;DR

CurvSSL introduces a curvature-based regularizer to self-supervised learning, enhancing local data manifold geometry understanding and improving representation quality on benchmark datasets.

Contribution

The paper proposes a novel curvature regularization method for SSL, extending it with an RKHS-based approach to better capture local data geometry.

Findings

CurvSSL achieves competitive or superior linear evaluation accuracy on MNIST and CIFAR-10.

The curvature regularizer improves local geometry alignment across augmentations.

Explicitly modeling local manifold bending complements existing statistical SSL methods.

Abstract

Self-supervised learning (SSL) has recently advanced through non-contrastive methods that couple an invariance term with variance, covariance, or redundancy-reduction penalties. While such objectives shape first- and second-order statistics of the representation, they largely ignore the local geometry of the underlying data manifold. In this paper, we introduce CurvSSL, a curvature-regularized self-supervised learning framework, and its RKHS extension, kernel CurvSSL. Our approach retains a standard two-view encoder-projector architecture with a Barlow Twins-style redundancy-reduction loss on projected features, but augments it with a curvature-based regularizer. Each embedding is treated as a vertex whose $k$ nearest neighbors define a discrete curvature score via cosine interactions on the unit hypersphere; in the kernel variant, curvature is computed from a normalized local Gram matrix in an RKHS. These scores are aligned and decorrelated across augmentations by a Barlow-style loss on a curvature-derived matrix, encouraging both view invariance and consistency of local manifold bending. Experiments on MNIST and CIFAR-10 datasets with a ResNet-18 backbone show that curvature-regularized SSL yields competitive or improved linear evaluation performance compared to Barlow Twins and VICReg. Our results indicate that explicitly shaping local geometry is a simple and effective complement to purely statistical SSL regularizers.