MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning

TL;DR

MiLoRA is a novel LLM finetuning method that updates only the minor singular components of weight matrices, preserving pretrained knowledge and improving performance across various benchmarks.

Contribution

This paper introduces MiLoRA, a new approach that selectively updates minor singular components, enhancing parameter efficiency and knowledge preservation during LLM finetuning.

Findings

MiLoRA outperforms existing methods on multiple benchmarks.

It effectively preserves pretrained knowledge during finetuning.

The approach is efficient and maintains model performance.

Abstract

Efficient finetuning of large language models (LLMs) aims to adapt the LLMs with reduced computational and memory cost. Previous LoRA-based approaches initialize the low-rank matrices with Gaussian distribution and zero values while keeping the original weight matrices frozen. However, the trainable model parameters optimized in an unguided subspace might interfere with the well-learned subspace of the pretrained weight matrices. In this paper, we propose MiLoRA, a simple yet effective LLM finetuning approach that only updates the minor singular components of the weight matrix while keeping the principal singular components frozen. It is observed that the minor matrix corresponds to the noisy or long-tail information, while the principal matrix contains important knowledge. The MiLoRA initializes the low-rank matrices within a subspace that is orthogonal to the principal matrix, thus the pretrained knowledge is expected to be well preserved. During finetuning, MiLoRA makes the most use of the less-optimized subspace for learning the labeled dataset. Extensive experiments on commonsense reasoning, math reasoning, instruction following and visual instruction following benchmarks present the superior performance of our method.