LoRA-Ensemble: Efficient Uncertainty Modelling for Self-Attention Networks

TL;DR

LoRA-Ensemble introduces a parameter-efficient implicit ensembling method for self-attention networks that improves uncertainty calibration and accuracy, reducing computational costs compared to explicit ensembles.

Contribution

It extends Low-Rank Adaptation (LoRA) into an implicit ensembling scheme for transformers, outperforming existing implicit methods and matching explicit ensemble performance.

Findings

Outperforms state-of-the-art implicit ensembling methods

Matches or exceeds explicit ensemble accuracy

Achieves superior calibration with lower computational cost

Abstract

Numerous real-world decisions rely on machine learning algorithms and require calibrated uncertainty estimates. However, modern methods often yield overconfident, uncalibrated predictions. The dominant approach to quantifying the uncertainty inherent in the model is to train an ensemble of separate predictors and measure their empirical variance. In an explicit implementation, the ensemble has a high computational cost and memory footprint, especially if the base model itself is already large, like modern transformers. This motivates efforts to develop implicit ensemble methods that emulate the ensemble without explicitly instantiating all its members. We introduce LoRA-Ensemble, a parameter-efficient ensembling method for self-attention networks. It is based on Low-Rank Adaptation (LoRA), originally developed for efficient LLM fine-tuning, and extends it into an implicit ensembling scheme, where all ensemble members share the same, pre-trained self-attention network, but have individual low-rank matrices for the attention projections. The resulting method not only outperforms state-of-the-art implicit techniques like BatchEnsemble, but even matches or exceeds the accuracy of an Explicit Ensemble, while at the same time achieving superior calibration.