GRASP: GRouped Activation Shared Parameterization for Parameter-Efficient Fine-Tuning and Robust Inference of Transformers

TL;DR

GRASP introduces a grouped activation sharing approach for parameter-efficient fine-tuning of transformers, significantly reducing trainable parameters while maintaining performance, and StochGRASP enhances robustness by modeling weight variability for deployment on edge hardware.

Contribution

The paper proposes GRASP, a novel grouped parameter sharing method for PEFT, and StochGRASP, a probabilistic extension that improves robustness against hardware noise.

Findings

GRASP matches or exceeds existing PEFT methods on GLUE and E2E NLG tasks.

StochGRASP outperforms deterministic variants under noise conditions.

Achieves an order of magnitude reduction in trainable parameters.

Abstract

Parameter-efficient fine-tuning (PEFT) provides a scalable alternative to full-model adaptation by updating only a small subset of parameters in large pre-trained models. We introduce GRASP - GRouped Activation Shared Parameterization - a lightweight PEFT framework that partitions the D-dimensional token representations of selected layers into K << D groups and learns a shared scaling and shifting vector for each group. This grouped modulation reduces the number of trainable parameters significantly while preserving the ability of the model to learn task-specific features. Building on this formulation, we further propose StochGRASP, which learns Gaussian distributions as perturbations to the pre-trained weights rather than deterministic values. This probabilistic parameterization along with a noise-aware loss function formulation enables modelling hardware-level variability in programmed weights and significantly improves robustness under non-ideal inference conditions-an important requirement for deployment on edge-based emerging AI hardware. Across GLUE (RoBERTa-base & RoBERTa-large) and E2E NLG (GPT-2 Medium), GRASP matches or exceeds the performance of established PEFT methods while achieving an order of magnitude reduction in trainable parameters compared to LoRA and BitFit. Under varying levels of noise, StochGRASP consistently outperforms deterministic variants, demonstrating its suitability for energy-efficient and noise-prone hardware platforms.