BERT-JEPA: Reorganizing CLS Embeddings for Language-Invariant Semantics

TL;DR

BERT-JEPA enhances BERT models with a JEPA-based training objective to create language-invariant embeddings, improving multilingual performance by reorganizing CLS embeddings into a language-agnostic space.

Contribution

This paper introduces BERT-JEPA, a novel training paradigm that integrates JEPA with BERT to produce language-invariant CLS embeddings, addressing collapse issues.

Findings

Improved multilingual benchmark performance

Reorganized CLS embeddings into language-agnostic space

Demonstrated effectiveness of JEPA in language models

Abstract

Joint Embedding Predictive Architectures (JEPA) are a novel self supervised training technique that have shown recent promise across domains. We introduce BERT-JEPA (BEPA), a training paradigm that adds a JEPA training objective to BERT-style models, working to combat a collapsed [CLS] embedding space and turning it into a language-agnostic space. This new structure leads to increased performance across multilingual benchmarks.