PiCa: Parameter-Efficient Fine-Tuning with Column Space Projection

TL;DR

PiCa is a new theoretically grounded parameter-efficient fine-tuning method that projects gradients onto the principal column space of pre-trained weights, improving adaptation and outperforming baselines.

Contribution

Introduces PiCa, a novel PEFT approach with a solid theoretical foundation and a weight-sharing strategy, enhancing fine-tuning efficiency across NLP and vision tasks.

Findings

PiCa outperforms state-of-the-art baselines with fewer parameters.

Gradient projection onto principal column space provides effective inductive bias.

PiCa demonstrates strong empirical results across diverse tasks.

Abstract

Fine-tuning large foundation models is essential for building expert models tailored to specialized tasks and domains, but fully updating billions of parameters is computationally prohibitive. Reducing the number of trainable parameters using Parameter-Efficient Fine-Tuning (PEFT), such as Low-Rank Adaptation (LoRA), is therefore crucial not only to reduce training costs but also to mitigate storage, caching, and serving overheads during deployment. Prior works, such as Singular Vectors-guided Fine-Tuning (SVFT), have shown that exploiting the geometry of pre-trained weights based on Singular Value Decomposition (SVD) can significantly improve parameter-efficiency, but they lack a solid theoretical foundation. In this paper, we introduce Parameter-Efficient Fine-Tuning with Column Space Projection (PiCa), a novel theoretically grounded PEFT method. We prove that projecting gradients onto the principal column space of pre-trained weights provides an effective inductive bias for adaptation and further enhance parameter efficiency through a novel weight-sharing strategy. Across diverse NLP and vision tasks, PiCa consistently outperforms state-of-the-art baselines under comparable or smaller parameter budgets, demonstrating both theoretical rigor and practical effectiveness.