Adaptive Principal Components Allocation with the $\ell_{2,g}$-regularized Gaussian Graphical Model for Efficient Fine-Tuning Large Models

TL;DR

This paper introduces a novel Gaussian Graphical Model-based parameter-efficient fine-tuning method for large models, effectively selecting critical parameters with fewer trainable parameters, demonstrated on the GLUE benchmark.

Contribution

It is the first to apply Gaussian Graphical Models to parameter-efficient fine-tuning, utilizing the $ ext{l}_{2,g}$-norm for parameter selection and global dependency capture.

Findings

Achieves competitive performance with fewer trainable parameters.

Demonstrates effectiveness on the GLUE benchmark.

Uses an efficient Block Coordinate Descent algorithm.

Abstract

In this work, we propose a novel Parameter-Efficient Fine-Tuning (PEFT) approach based on Gaussian Graphical Models (GGMs), marking the first application of GGMs to PEFT tasks, to the best of our knowledge. The proposed method utilizes the $\ell_{2,g}$-norm to effectively select critical parameters and capture global dependencies. The resulting non-convex optimization problem is efficiently solved using a Block Coordinate Descent (BCD) algorithm. Experimental results on the GLUE benchmark [24] for fine-tuning RoBERTa-Base [18] demonstrate the effectiveness of the proposed approach, achieving competitive performance with significantly fewer trainable parameters. The code for this work is available at: https://github.com/jzheng20/Course projects.git.