Joint-Centric Dual Contrastive Alignment with Structure-Preserving and Information-Balanced Regularization

TL;DR

HILBERT is a multimodal framework that uses reciprocal contrastive training and regularization to learn structured, balanced audio-text document representations from long sequences, improving performance in low-resource, imbalanced tasks.

Contribution

It introduces a reciprocal dual contrastive objective and regularizers for stable long-sequence multimodal embedding, advancing cross-modal alignment under severe imbalance.

Findings

HILBERT achieves superior performance on imbalanced multi-class audio-text tasks.

The reciprocal contrastive objective effectively aligns audio and text modalities.

Regularizers stabilize long-sequence fusion and preserve structural consistency.

Abstract

We propose HILBERT (HIerarchical Long-sequence Balanced Embedding with Reciprocal contrastive Training), a cross-attentive multimodal framework for learning document-level audio-text representations from long, segmented sequences in low-resource data settings. HILBERT leverages frozen pre-trained speech and language encoders to extract segment-level features, which are aggregated via cross-modal attention and self-attentive pooling to form modality-specific document representations and a joint cross-attentive embedding. To align modalities while preserving modality-specific structure under severe audio-text dimensional imbalance, we introduce a reciprocal dual contrastive objective that simultaneously aligns audio-to-joint and text-to-joint representations, rather than directly contrasting audio and text alone. Two auxiliary regularizers further stabilize long-sequence fusion: a Centered Kernel Alignment (CKA) loss that preserves structural consistency between each modality and the joint embedding, and a mutual information balancing loss that prevents dominance of a single modality by equalizing information flow from audio and text into the joint space. For downstream prediction, HILBERT employs a Mixture-of-Experts (MoE) classifier over concatenated audio, text, and joint representations to accommodate heterogeneous label regimes. Extensive evaluation across multiple audio-text backbone combinations demonstrates that HILBERT learns semantically meaningful long-sequence representations and achieves superior performance on highly imbalanced multi-class settings.