Mining Attribute Subspaces for Efficient Fine-tuning of 3D Foundation Models

TL;DR

This paper investigates the structure of LoRA subspaces in 3D foundation models, demonstrating that disentangled subspaces associated with different variations can be combined to improve fine-tuning efficiency and accuracy.

Contribution

It introduces a method to extract and combine disentangled LoRA subspaces from synthetic data for more efficient and accurate 3D model fine-tuning.

Findings

Disentangled LoRA subspaces correspond to different data variations.

Combining subspaces improves fine-tuning efficiency and accuracy.

Synthetic data-based subspaces generalize to real datasets.

Abstract

With the emergence of 3D foundation models, there is growing interest in fine-tuning them for downstream tasks, where LoRA is the dominant fine-tuning paradigm. As 3D datasets exhibit distinct variations in texture, geometry, camera motion, and lighting, there are interesting fundamental questions: 1) Are there LoRA subspaces associated with each type of variation? 2) Are these subspaces disentangled (i.e., orthogonal to each other)? 3) How do we compute them effectively? This paper provides answers to all these questions. We introduce a robust approach that generates synthetic datasets with controlled variations, fine-tunes a LoRA adapter on each dataset, and extracts a LoRA sub-space associated with each type of variation. We show that these subspaces are approximately disentangled. Integrating them leads to a reduced LoRA subspace that enables efficient LoRA fine-tuning with improved prediction accuracy for downstream tasks. In particular, we show that such a reduced LoRA subspace, despite being derived entirely from synthetic data, generalizes to real datasets. An ablation study validates the effectiveness of the choices in our approach.