From Alignment to Prediction: A Study of Self-Supervised Learning and Predictive Representation Learning

TL;DR

This paper introduces Predictive Representation Learning (PRL) as a new paradigm in self-supervised learning, emphasizing latent prediction of unobserved data components, and compares it with existing alignment and reconstruction methods.

Contribution

It defines PRL, proposes a taxonomy including PRL, alignment, and reconstruction, and analyzes architectures like I-JEPA, providing theoretical insights and empirical comparisons.

Findings

MAE achieves perfect similarity of 1.00

BYOL and I-JEPA attain high accuracy of 0.98 and 0.95

I-JEPA shows strong robustness scores of 0.78

Abstract

Self-supervised learning has emerged as a major technique for the task of learning from unlabeled data, where the current methods mostly revolve around alignment of representations and input recon struction. Although such approaches have demonstrated excellent performance in practice, their scope remains mostly confined to learning from observed data and does not provide much help in terms of a learning structure that is predictive of the data distribution. In this paper, we study some of the recent developments in the realm of self-supervised learning. We define a new category called Predictive Representation Learning (PRL), which revolves around the latent prediction of unobserved components of data based on the observation. We propose a common taxonomy that classifies PRL along with alignment and reconstruction-based learning approaches. Furthermore, we argue that Joint-Embedding Predictive Architecture(JEPA) can be considered as an exemplary member of this new paradigm. We further discuss theoretical perspectives and open challenges, highlighting predictive representation learning as a promising direction for future self-supervised learning research. In this study, we implemented Bootstrap Your Own Latent (BYOL), Masked Autoencoders (MAE), and Image-JEPA (I-JEPA) for comparative analysis. The results indicate that MAE achieves perfect similarity of 1.00, but exhibits relatively weak robustness of 0.55. In contrast, BYOL and I-JEPA attain accuracies of 0.98 and 0.95, with robustness scores of 0.75 and 0.78, respectively.