Cross-Modal Bayesian Low-Rank Adaptation for Uncertainty-Aware Multimodal Learning

TL;DR

CALIBER is a Bayesian low-rank adaptation framework for multimodal learning that enhances uncertainty estimation by conditioning on cross-modal attention, improving performance in low-resource audio-text tasks.

Contribution

It introduces a novel Bayesian PEFT method that uses cross-attention to condition the adaptation process, enabling uncertainty-aware multimodal learning with efficiency.

Findings

CALIBER matches or outperforms existing baselines in diverse tasks.

Token-level cross-attention provides the most consistent performance gains.

The framework effectively estimates heteroscedastic uncertainty in multimodal settings.

Abstract

Large pre-trained language models are increasingly adapted to downstream tasks using parameter-efficient fine-tuning (PEFT), but existing PEFT methods are typically deterministic and unimodal, making them poorly suited for low-resource multimodal settings where predictive uncertainty and cross-modal reliability both matter. We introduce CALIBER (Context-Aware Low-rank Inference with Bayesian Embedding Regularization), a multimodal uncertainty-aware PEFT framework for audio-text learning. CALIBER extends Bayesian low-rank adaptation by conditioning the variational posterior in the adapter space on per-layer, token-level text-audio cross-attention. Specifically, text-derived low-rank features attend to frame-level audio embeddings to produce localized acoustic context, which then modulates the mean and variance of a compact stochastic latent matrix within the rank-$r$ adapter space. This design treats audio not only as an additional feature source, but as a contextual reliability signal that shapes both adaptation and confidence. By confining stochasticity to a low-dimensional latent component, CALIBER retains the computational efficiency and scalability of PEFT while enabling heteroscedastic multimodal uncertainty estimation. Experimental results across diverse text and audio backbones show that CALIBER consistently matches or improves upon text-only Bayesian PEFT and conventional multimodal transfer-learning baselines, with token-level cross-attention yielding the most consistent gains. Our findings demonstrate that localized cross-modal conditioning is an effective and lightweight mechanism for uncertainty-aware multimodal adaptation.