Maintaining Structural Integrity in Parameter Spaces for Parameter Efficient Fine-tuning

TL;DR

This paper introduces a generalized low-rank tensor adaptation framework for parameter-efficient fine-tuning that preserves the structural integrity of high-dimensional parameter spaces across diverse AI models.

Contribution

It proposes a novel method that models changes in high-dimensional parameter spaces via a low-rank core, maintaining topological structure during fine-tuning.

Findings

Effective across vision, NLP, and multi-modal tasks

Outperforms existing fine-tuning methods in resource efficiency

Preserves structural integrity of parameter spaces

Abstract

Adapting pre-trained foundation models for various downstream tasks has been prevalent in artificial intelligence. Due to the vast number of tasks and high costs, adjusting all parameters becomes unfeasible. To mitigate this, several fine-tuning techniques have been developed to update the pre-trained model weights in a more resource-efficient manner, such as through low-rank adjustments. Yet, almost all of these methods focus on linear weights, neglecting the intricacies of parameter spaces in higher dimensions like 4D. Alternatively, some methods can be adapted for high-dimensional parameter space by compressing changes in the original space into two dimensions and then employing low-rank matrix adaptations. However, these approaches destructs the structural integrity of the involved high-dimensional spaces. To tackle the diversity of dimensional spaces across different foundation models and provide a more precise representation of the changes within these spaces, this paper introduces a generalized parameter-efficient fine-tuning framework, designed for various dimensional parameter space. Specifically, our method asserts that changes in each dimensional parameter space are based on a low-rank core space which maintains the consistent topological structure with the original space. It then models the changes through this core space alongside corresponding weights to reconstruct alterations in the original space. It effectively preserves the structural integrity of the change of original N-dimensional parameter space, meanwhile models it via low-rank tensor adaptation. Extensive experiments on computer vision, natural language processing and multi-modal tasks validate the effectiveness of our method.