Sketch to Adapt: Fine-Tunable Sketches for Efficient LLM Adaptation

TL;DR

SketchTune introduces a novel data compression-based adaptation method for large language models that improves efficiency and performance by using sketching techniques instead of traditional low-rank assumptions.

Contribution

It proposes SketchTune, a unified framework that compresses LLM weights into fine-tunable sketches, enabling faster, more memory-efficient adaptation without complex two-path computation.

Findings

Outperforms existing PEFT methods on diverse tasks.

Uses significantly smaller models with comparable or better accuracy.

Achieves 14.48% higher accuracy on GSM8K with fewer trainable parameters.

Abstract

Adapting pre-trained large language models (LLMs) is crucial but challenging due to their enormous size. Parameter-efficient fine-tuning (PEFT) techniques typically employ additive adapters applied to frozen model weights. To further reduce memory usage, model weights are often compressed through quantization. However, existing PEFT methods often yield suboptimal model quality because they rely on restrictive assumptions, such as low-rank constraints on adapters to limit the number of trainable parameters. We find that sketching, a popular data compression technique, can serve as an efficient LLM adaptation strategy while avoiding the low-rank assumption. We introduce SketchTune, a compressive adaptation strategy that compresses LLM weights into compact fine-tunable sketches, integrating compression and adaptation into a unified framework. This integration eliminates the need for complex two-path computation in existing PEFT techniques, enabling faster and more memory-efficient training and inference. SketchTune is supported by mathematical insights into matrix classes that are better approximated using sketching rather than low-rank methods. Our extensive evaluations with Llama and Mistral models demonstrate that SketchTune outperforms leading PEFT methods across diverse tasks while using substantially smaller base models and comparable trainable parameters. As a highlight, SketchTune outperforms LoRA, DoRA, and S2FT on commonsense and math benchmarks using 2.6-3.5$\times$ smaller base models and exceeds LoftQ in accuracy by 14.48% on GSM8K with 7.3$\times$ fewer trainable parameters. Our code is available at https://github.com/LeanModels/SketchTune.