Train Less, Infer Faster: Efficient Model Finetuning and Compression via Structured Sparsity

TL;DR

This paper introduces a sparsification-based finetuning method for large language models that reduces parameters and inference time while maintaining accuracy, supported by theoretical guarantees and empirical results.

Contribution

It presents a novel sparsification scheme with stochastic gates for efficient finetuning, outperforming recent methods in efficiency and performance.

Findings

Reduces 20-40% of model parameters without accuracy loss

Outperforms recent finetuning baselines in efficiency and performance

Provides theoretical convergence guarantees for the sparsification process

Abstract

Fully finetuning foundation language models (LMs) with billions of parameters is often impractical due to high computational costs, memory requirements, and the risk of overfitting. Although methods like low-rank adapters help address these challenges by adding small trainable modules to the frozen LM, they also increase memory usage and do not reduce inference latency. We uncover an intriguing phenomenon: sparsifying specific model rows and columns enables efficient task adaptation without requiring weight tuning. We propose a scheme for effective finetuning via sparsification using training stochastic gates, which requires minimal trainable parameters, reduces inference time, and removes 20--40\% of model parameters without significant accuracy loss. Empirical results show it outperforms recent finetuning baselines in efficiency and performance. Additionally, we provide theoretical guarantees for the convergence of this stochastic gating process, and show that our method admits a simpler and better-conditioned optimization landscape compared to LoRA. Our results highlight sparsity as a compelling mechanism for task-specific adaptation in LMs.