LoRA3D: Low-Rank Self-Calibration of 3D Geometric Foundation Models

TL;DR

LoRA3D introduces an efficient self-calibration method for 3D geometric models that refines predictions and adapts models to new scenes using only multi-view images, achieving significant performance gains without external data.

Contribution

The paper presents LoRA3D, a novel self-calibration pipeline that specializes pre-trained 3D models to target scenes using multi-view predictions and low-rank adaptation, requiring minimal computation and no manual labels.

Findings

Achieves up to 88% performance improvement in 3D tasks.

Completes calibration on a single GPU in 5 minutes.

Requires only 18MB per low-rank adapter.

Abstract

Emerging 3D geometric foundation models, such as DUSt3R, offer a promising approach for in-the-wild 3D vision tasks. However, due to the high-dimensional nature of the problem space and scarcity of high-quality 3D data, these pre-trained models still struggle to generalize to many challenging circumstances, such as limited view overlap or low lighting. To address this, we propose LoRA3D, an efficient self-calibration pipeline to $\textit{specialize}$ the pre-trained models to target scenes using their own multi-view predictions. Taking sparse RGB images as input, we leverage robust optimization techniques to refine multi-view predictions and align them into a global coordinate frame. In particular, we incorporate prediction confidence into the geometric optimization process, automatically re-weighting the confidence to better reflect point estimation accuracy. We use the calibrated confidence to generate high-quality pseudo labels for the calibrating views and use low-rank adaptation (LoRA) to fine-tune the models on the pseudo-labeled data. Our method does not require any external priors or manual labels. It completes the self-calibration process on a $\textbf{single standard GPU within just 5 minutes}$. Each low-rank adapter requires only $\textbf{18MB}$ of storage. We evaluated our method on $\textbf{more than 160 scenes}$ from the Replica, TUM and Waymo Open datasets, achieving up to $\textbf{88% performance improvement}$ on 3D reconstruction, multi-view pose estimation and novel-view rendering.