Enhancing VICReg: Random-Walk Pairing for Improved Generalization and Better Global Semantics Capturing

TL;DR

This paper introduces SAG-VICReg, an improved self-supervised learning method that enhances global semantic capturing and generalization by building on VICReg, supported by new training techniques and a novel evaluation metric.

Contribution

It proposes SAG-VICReg, a novel extension of VICReg, with training enhancements and a new embedding evaluation metric for better global semantics and robustness.

Findings

SAG-VICReg outperforms existing SSL methods on global semantic metrics.

The method maintains competitive local evaluation results.

The new metric effectively assesses global data structure without labels.

Abstract

In this paper, we argue that viewing VICReg-a popular self-supervised learning (SSL) method--through the lens of spectral embedding reveals a potential source of sub-optimality: it may struggle to generalize robustly to unseen data due to overreliance on the training data. This observation invites a closer look at how well this method achieves its goal of producing meaningful representations of images outside of the training set as well. Here, we investigate this issue and introduce SAG-VICReg (Stable and Generalizable VICReg), a method that builds on VICReg by incorporating new training techniques. These enhancements improve the model's ability to capture global semantics within the data and strengthen the generalization capabilities. Experiments demonstrate that SAG-VICReg effectively addresses the generalization challenge while matching or surpassing diverse state-of-the-art SSL baselines. Notably, our method exhibits superior performance on metrics designed to evaluate global semantic understanding, while simultaneously maintaining competitive results on local evaluation metrics. Furthermore, we propose a new standalone evaluation metric for embeddings that complements the standard evaluation methods and accounts for the global data structure without requiring labels--a key issue when tagged data is scarce or not available.