SineLoRA$\Delta$: Sine-Activated Delta Compression

TL;DR

SineLoRAΔ introduces a sinusoidal activation to enhance delta compression of models, enabling significant memory savings while maintaining performance across multiple domains.

Contribution

It extends low-rank adapters with sinusoidal functions for better expressivity in quantized settings, supported by theoretical analysis of stable rank evolution.

Findings

Achieves up to 66% memory reduction with comparable performance.

Validates across language, vision-language, and image generation tasks.

Provides a novel application of the Bj{}ntegaard Delta metric for adapter comparison.

Abstract

Resource-constrained weight deployment is a task of immense practical importance. Recently, there has been interest in the specific task of \textit{Delta Compression}, where parties each hold a common base model and only communicate compressed weight updates. However, popular parameter efficient updates such as Low Rank Adaptation (LoRA) face inherent representation limitations - which are especially pronounced when combined with aggressive quantization. To overcome this, we build on recent work that improves LoRA representation capacity by using fixed-frequency sinusoidal functions to increase stable rank without adding additional parameters. We extend this to the quantized setting and present the first theoretical analysis showing how stable rank evolves under quantization. From this, we introduce SineLoRA$\Delta$, a principled and effective method for delta compression that improves the expressivity of quantized low-rank adapters by applying a sinusoidal activation. We validate SineLoRA$\Delta$ across a diverse variety of domains - including language modeling, vision-language tasks, and text-to-image generation - achieving up to 66% memory reduction with similar performance. We additionally provide a novel application of the canonical Bj{\o}ntegaard Delta metric to consistently compare adapter compression changes across the rate-distortion curve.