CoSA: Compressed Sensing-Based Adaptation of Large Language Models

TL;DR

CoSA introduces a novel PEFT method based on compressed sensing theory, allowing more expressive and efficient adaptation of large language models without low-rank constraints, outperforming existing methods across diverse tasks.

Contribution

The paper proposes CoSA, a PEFT approach using fixed random projections and a learnable core, extending compressed sensing theory to improve model adaptation expressivity.

Findings

CoSA matches or outperforms state-of-the-art PEFT methods.

Effective across 10 diverse NLP tasks and multiple model scales.

Provides a theoretical foundation for efficient model adaptation.

Abstract

Parameter-Efficient Fine-Tuning (PEFT) has emerged as a practical paradigm for adapting large language models (LLMs) without updating all parameters. Most existing approaches, such as LoRA and PiSSA, rely on low-rank decompositions of weight updates. However, the low-rank assumption may restrict expressivity, particularly in task-specific adaptation scenarios where singular values are distributed relatively uniformly. To address this limitation, we propose CoSA (Compressed Sensing-Based Adaptation), a new PEFT method extended from compressed sensing theory. Instead of constraining weight updates to a low-rank subspace, CoSA expresses them through fixed random projection matrices and a compact learnable core. We provide a formal theoretical analysis of CoSA as a synthesis process, proving that weight updates can be compactly encoded into a low-dimensional space and mapped back through random projections. Extensive experimental results show that CoSA provides a principled perspective for efficient and expressive multi-scale model adaptation. Specifically, we evaluate CoSA on 10 diverse tasks, including natural language understanding and generation, employing 5 models of different scales from RoBERTa, Llama, and Qwen families. Across these settings, CoSA consistently matches or outperforms state-of-the-art PEFT methods.