The Scaling Law for LoRA Base on Mutual Information Upper Bound

TL;DR

This paper introduces a mutual information upper bound metric to better understand and predict the performance scaling law of LoRA fine-tuning in large models, outperforming traditional external metrics.

Contribution

The paper proposes a novel internal metric based on MIUB theory to analyze the scaling law of LoRA fine-tuning, validated on large models and benchmark datasets.

Findings

MIUB metric aligns more accurately with the scaling law

Outperforms cross-entropy and perplexity in stability

Validated on Llama3-8B and Phi3-3B models

Abstract

LoRA (Low-Rank Adaptation) is a widely used model fine-tuning method. In fine-tuning, the law among model performance, model parameters, and data complexity has been a focal issue in the field. Existing methods often leverage external metrics (such as cross-entropy or perplexity) to evaluate model performance. In the fine-tuning process for large models, two types of knowledge are typically involved: the frozen, general knowledge acquired by the model during pre-training and the new knowledge learned through the LoRA module from the current data. Generally, the less LoRA's learned knowledge relies on the large model, the more it captures the specific knowledge of new data, thereby enhancing its adaptability to new tasks. However, external metrics do not readily capture the dependency relationship between these two types of knowledge. Therefore, we designed an internal metric based on the Mutual Information Upper Bound (MIUB) theory to investigate the scaling law of large-model LoRA fine-tuning. In our experiments, we validated this approach on benchmark datasets, using the Llama3-8B and Phi3-3B models. The results show that the proposed MIUB metric aligns more accurately and stably with the scaling law of LoRA fine-tuning compared to cross-entropy and perplexity.