GeRA: Label-Efficient Geometrically Regularized Alignment

TL;DR

GeRA is a semi-supervised method that improves the alignment of pretrained unimodal encoders using geometric regularization, reducing the need for large paired datasets and preserving local manifold structures.

Contribution

We introduce a modality-agnostic geometric regularization technique that enhances alignment of pretrained encoders with limited paired data by leveraging manifold geometry.

Findings

Significant improvement in speech-text and image-text alignment quality.

Effective with small amounts of paired data due to geometric regularization.

Outperforms leading baselines in alignment tasks.

Abstract

Pretrained unimodal encoders incorporate rich semantic information into embedding space structures. To be similarly informative, multi-modal encoders typically require massive amounts of paired data for alignment and training. We introduce a semi-supervised Geometrically Regularized Alignment (GeRA) method to align the embedding spaces of pretrained unimodal encoders in a label-efficient way. Our method leverages the manifold geometry of unpaired (unlabeled) data to improve alignment performance. To prevent distortions to local geometry during the alignment process, potentially disrupting semantic neighborhood structures and causing misalignment of unobserved pairs, we introduce a geometric loss term. This term is built upon a diffusion operator that captures the local manifold geometry of the unimodal pretrained encoders. GeRA is modality-agnostic and thus can be used to align pretrained encoders from any data modalities. We provide empirical evidence to the effectiveness of our method in the domains of speech-text and image-text alignment. Our experiments demonstrate significant improvement in alignment quality compared to a variaty of leading baselines, especially with a small amount of paired data, using our proposed geometric regularization.