Efficiently Seeking Flat Minima for Better Generalization in Fine-Tuning Large Language Models and Beyond

TL;DR

This paper introduces FMLoRA and EFMLoRA, methods to find flat minima in low-rank adaptation for large models, improving generalization and efficiency in fine-tuning across NLP and vision-language tasks.

Contribution

The paper proposes a novel approach to seek flat minima in LoRA, with theoretical transferability of perturbations and an efficient variant, enhancing model generalization and fine-tuning performance.

Findings

EFMLoRA matches LoRA's efficiency while outperforming it in accuracy.

EFMLoRA surpasses full fine-tuning on large language models.

Sharpness correlates with LoRA's generalization, verified empirically.

Abstract

Little research explores the correlation between the expressive ability and generalization ability of the low-rank adaptation (LoRA). Sharpness-Aware Minimization (SAM) improves model generalization for both Convolutional Neural Networks (CNNs) and Transformers by encouraging convergence to locally flat minima. However, the connection between sharpness and generalization has not been fully explored for LoRA due to the lack of tools to either empirically seek flat minima or develop theoretical methods. In this work, we propose Flat Minima LoRA (FMLoRA) and its efficient version, i.e., EFMLoRA, to seek flat minima for LoRA. Concretely, we theoretically demonstrate that perturbations in the full parameter space can be transferred to the low-rank subspace. This approach eliminates the potential interference introduced by perturbations across multiple matrices in the low-rank subspace. Our extensive experiments on large language models and vision-language models demonstrate that EFMLoRA achieves optimize efficiency comparable to that of LoRA while simultaneously attaining comparable or even better performance. For example, on the GLUE dataset with RoBERTa-large, EFMLoRA outperforms LoRA and full fine-tuning by 1.0% and 0.5% on average, respectively. On vision-language models, e.g., Qwen-VL-Chat, there are performance improvements of 1.5% and 1.0% on the SQA and VizWiz datasets, respectively. These empirical results also verify that the generalization of LoRA is closely related to sharpness, which is omitted by previous methods.