GradOT: Training-free Gradient-preserving Offsite-tuning for Large Language Models

TL;DR

GradOT introduces a training-free, gradient-preserving offsite-tuning method for large language models that enhances privacy and performance by selectively compressing models without additional training.

Contribution

It presents a novel gradient-preserving compression approach for offsite-tuning, providing theoretical analysis and practical benefits over existing methods.

Findings

Outperforms existing OT methods in privacy and performance

Uses gradient-preserving compression techniques like rank compression and channel pruning

Offers a training-free, theoretically grounded solution for LLM offsite-tuning

Abstract

The rapid growth of large language models (LLMs) with traditional centralized fine-tuning emerges as a key technique for adapting these models to domain-specific challenges, yielding privacy risks for both model and data owners. One promising solution, called offsite-tuning (OT), is proposed to address these challenges, where a weaker emulator is compressed from the original model and further fine-tuned with adapter to enhance privacy. However, the existing OT-based methods require high computational costs and lack theoretical analysis. This paper introduces a novel OT approach based on gradient-preserving compression, named GradOT. By analyzing the OT problem through the lens of optimization, we propose a method that selectively applies compression techniques such as rank compression and channel pruning, preserving the gradients of fine-tuned adapters while ensuring privacy. Extensive experiments demonstrate that our approach surpasses existing OT methods, both in terms of privacy protection and model performance. Our method provides a theoretical foundation for OT and offers a practical, training-free solution for offsite-tuning of large-scale LLMs.