C-LoRA: Contextual Low-Rank Adaptation for Uncertainty Estimation in Large Language Models

TL;DR

C-LoRA introduces a novel, input-specific, lightweight fine-tuning method for large language models that improves uncertainty estimation and robustness in few-shot settings by dynamically adapting to each data sample.

Contribution

It develops a new contextual LoRA module that enhances uncertainty calibration and model robustness, addressing limitations of existing methods in data-scarce scenarios.

Findings

C-LoRA outperforms state-of-the-art uncertainty-aware LoRA methods in experiments.

It achieves well-calibrated uncertainties and robust predictions.

Ablation studies confirm the importance of contextual modules.

Abstract

Low-Rank Adaptation (LoRA) offers a cost-effective solution for fine-tuning large language models (LLMs), but it often produces overconfident predictions in data-scarce few-shot settings. To address this issue, several classical statistical learning approaches have been repurposed for scalable uncertainty-aware LoRA fine-tuning. However, these approaches neglect how input characteristics affect the predictive uncertainty estimates. To address this limitation, we propose Contextual Low-Rank Adaptation (C-LoRA) as a novel uncertainty-aware and parameter efficient fine-tuning approach, by developing new lightweight LoRA modules contextualized to each input data sample to dynamically adapt uncertainty estimates. Incorporating data-driven contexts into the parameter posteriors, C-LoRA mitigates overfitting, achieves well-calibrated uncertainties, and yields robust predictions. Extensive experiments on LLaMA2-7B models demonstrate that C-LoRA consistently outperforms the state-of-the-art uncertainty-aware LoRA methods in both uncertainty quantification and model generalization. Ablation studies further confirm the critical role of our contextual modules in capturing sample-specific uncertainties. C-LoRA sets a new standard for robust, uncertainty-aware LLM fine-tuning in few-shot regimes. Although our experiments are limited to 7B models, our method is architecture-agnostic and, in principle, applies beyond this scale; studying its scaling to larger models remains an open problem. Our code is available at https://github.com/ahra99/c_lora.