Parameter-efficient Bayesian Neural Networks for Uncertainty-aware Depth Estimation

TL;DR

This paper explores combining parameter-efficient fine-tuning methods with Bayesian inference to improve uncertainty estimation and robustness in large-scale Transformer models for monocular depth estimation.

Contribution

It introduces a novel LoRA-inspired PEFT method, CoLoRA, and demonstrates its effectiveness in enhancing Bayesian inference for depth estimation tasks.

Findings

PEFT methods improve Bayesian depth estimation performance

Combining LoRA, BitFit, DiffFit, and CoLoRA enhances uncertainty quantification

Proposed methods outperform traditional approaches in reliability and robustness

Abstract

State-of-the-art computer vision tasks, like monocular depth estimation (MDE), rely heavily on large, modern Transformer-based architectures. However, their application in safety-critical domains demands reliable predictive performance and uncertainty quantification. While Bayesian neural networks provide a conceptually simple approach to serve those requirements, they suffer from the high dimensionality of the parameter space. Parameter-efficient fine-tuning (PEFT) methods, in particular low-rank adaptations (LoRA), have emerged as a popular strategy for adapting large-scale models to down-stream tasks by performing parameter inference on lower-dimensional subspaces. In this work, we investigate the suitability of PEFT methods for subspace Bayesian inference in large-scale Transformer-based vision models. We show that, indeed, combining BitFit, DiffFit, LoRA, and CoLoRA, a novel LoRA-inspired PEFT method, with Bayesian inference enables more robust and reliable predictive performance in MDE.