Non-Contrastive Vision-Language Learning with Predictive Embedding Alignment

TL;DR

NOVA introduces a non-contrastive vision-language alignment method that simplifies training by predicting text embeddings from images, eliminating the need for negative sampling and hyperparameter tuning, and achieves superior zero-shot classification results.

Contribution

It presents NOVA, a novel non-contrastive framework for vision-language learning that simplifies training and improves stability and performance over contrastive methods.

Findings

Outperforms standard baselines on zero-shot chest X-ray classification

Exhibits more consistent training runs

Reduces training complexity with a single hyperparameter

Abstract

Vision-language models have transformed multimodal representation learning, yet dominant contrastive approaches like CLIP require large batch sizes, careful negative sampling, and extensive hyperparameter tuning. We introduce NOVA, a NOn-contrastive Vision-language Alignment framework based on joint embedding prediction with distributional regularization. NOVA aligns visual representations to a frozen, domain-specific text encoder by predicting text embeddings from augmented image views, while enforcing an isotropic Gaussian structure via Sketched Isotropic Gaussian Regularization (SIGReg). This eliminates the need for negative sampling, momentum encoders, or stop-gradients, reducing the training objective to a single hyperparameter. We evaluate NOVA on zeroshot chest X-ray classification using ClinicalBERT as the text encoder and Vision Transformers trained from scratch on MIMIC-CXR. On zero-shot classification across three benchmark datasets, NOVA outperforms multiple standard baselines while exhibiting substantially more consistent training runs. Our results demonstrate that non-contrastive vision-language pretraining offers a simpler, more stable, and more effective alternative to contrastive methods.