Beyond Feature Fusion: Contextual Bayesian PEFT for Multimodal Uncertainty Estimation

TL;DR

CoCo-LoRA is a novel multimodal, uncertainty-aware parameter-efficient fine-tuning method that incorporates audio context to improve reliability in speech prediction tasks.

Contribution

It introduces a new Bayesian PEFT approach that models audio-driven uncertainty without high-dimensional fusion, enhancing multimodal prediction robustness.

Findings

CoCo-LoRA outperforms text-only PEFT and feature-fusion baselines on diverse tasks.

It effectively models heteroscedastic uncertainty driven by audio context.

The method maintains scalability while incorporating external acoustic information.

Abstract

We introduce CoCo-LoRA, a multimodal, uncertainty-aware parameter-efficient fine-tuning method for text prediction tasks accompanied by audio context. Existing PEFT approaches such as LoRA are efficient but typically deterministic, while recent Bayesian low-rank adapters model uncertainty in a lightweight way yet remain largely unimodal and condition uncertainty primarily on internal text features. This leaves them poorly equipped to reflect uncertainty driven by external acoustic factors such as background noise, channel variability, or speaking style, which can materially affect reliability in speech-centered applications. CoCo-LoRA addresses this gap by conditioning a contextual variational posterior in the low-rank space on both local text-derived adapter features and an audio-derived context signal. A pooled audio embedding is projected once into a shared context space and then adapted through lightweight layer-wise heads, enabling global-to-local, depth-specific modulation of the adapter uncertainty and update without high-dimensional multimodal fusion. Stochasticity is confined to a compact latent component in the rank space, preserving PEFT scalability while producing audio-sensitive, heteroscedastic uncertainty. Based on our evaluations across diverse tasks and backbone combinations, CoCo-LoRA consistently matches or outperforms text-only PEFT and conventional feature-fusion transfer baselines, particularly on high-coverage labels where reliable adaptation is critical. The results indicate that using audio as a contextual uncertainty signal, rather than as a fused feature stream, provides a robust and parameter-efficient alternative for multimodal low-resource prediction.