When Is Rank-1 Enough? Geometry-Guided Initialization for Parameter-Efficient Fine-Tuning

TL;DR

This paper introduces Gap-Init, a geometry-guided initialization method that aligns low-rank model updates with pretrained modality gaps, significantly improving the stability and performance of rank-1 fine-tuning in multimodal models.

Contribution

The paper identifies the cause of instability in rank-1 fine-tuning and proposes a novel initialization method that aligns updates with modality gaps, enhancing stability and effectiveness.

Findings

Gap-Init stabilizes rank-1 training across multiple tasks.

It matches or outperforms higher-rank baselines.

Alignment at initialization is crucial at low ranks.

Abstract

Parameter-efficient fine-tuning (PEFT) is a standard way to adapt multimodal large language models, yet extremely low-rank settings -- especially rank-1 LoRA -- are often unstable. We show that this instability is not solely due to limited capacity: in the rank-1 regime, optimization is highly sensitive to the update direction. Concretely, pretrained vision and text features form mismatched anisotropic regions, yielding a dominant "gap" direction that acts like a translation component and disproportionately steers early gradients under rank-1 constraints. Analyzing pretrained representations, we identify a modality-gap axis that dominates early gradient flow, while a random rank-1 initialization is unlikely to align with it, leading to weak gradients and training collapse. We propose Gap-Init, a geometry-aware initialization that aligns the rank-1 LoRA direction with an estimated modality-gap vector from a small calibration set, while keeping the initial LoRA update zero. Across multiple vision-language tasks and backbones, Gap-Init consistently stabilizes rank-1 training and can match or outperform strong rank-8 baselines. Our results suggest that at the extreme low-rank limit, initial alignment can matter as much as rank itself.