KerJEPA: Kernel Discrepancies for Euclidean Self-Supervised Learning

TL;DR

KerJEPA introduces kernel-based regularizers for Euclidean self-supervised learning, expanding kernel choices and priors to improve training stability and flexibility in joint-embedding predictive architectures.

Contribution

The paper proposes a new family of KerJEPAs with kernel-based regularizers, generalizing previous methods and providing closed-form solutions for high-dimensional sliced MMDs.

Findings

Improved training stability in self-supervised learning.

Enhanced flexibility through expanded kernel and prior choices.

Closed-form high-dimensional sliced MMD computations.

Abstract

Recent breakthroughs in self-supervised Joint-Embedding Predictive Architectures (JEPAs) have established that regularizing Euclidean representations toward isotropic Gaussian priors yields provable gains in training stability and downstream generalization. We introduce a new, flexible family of KerJEPAs, self-supervised learning algorithms with kernel-based regularizers. One instance of this family corresponds to the recently-introduced LeJEPA Epps-Pulley regularizer which approximates a sliced maximum mean discrepancy (MMD) with a Gaussian prior and Gaussian kernel. By expanding the class of viable kernels and priors and computing the closed-form high-dimensional limit of sliced MMDs, we develop alternative KerJEPAs with a number of favorable properties including improved training stability and design flexibility.