MSLoRA: Multi-Scale Low-Rank Adaptation via Attention Reweighting

TL;DR

MSLoRA is a versatile, parameter-efficient adapter that enhances various vision tasks by reweighting features across CNNs and ViTs, without altering the backbone weights.

Contribution

MSLoRA introduces a unified, lightweight adaptation module that works across architectures by combining low-rank projections with multi-scale attention reweighting.

Findings

Consistently improves transfer performance on multiple vision tasks.

Requires less than 5% of backbone parameters for adaptation.

Enables stable training and fast convergence across architectures.

Abstract

We introduce MSLoRA, a backbone-agnostic, parameter-efficient adapter that reweights feature responses rather than re-tuning the underlying backbone. Existing low-rank adaptation methods are mostly confined to vision transformers (ViTs) and struggle to generalize across architectures. MSLoRA unifies adaptation for both convolutional neural networks (CNNs) and ViTs by combining a low-rank linear projection with a multi-scale nonlinear transformation that jointly modulates spatial and channel attention. The two components are fused through pointwise multiplication and a residual connection, yielding a lightweight module that shifts feature attention while keeping pretrained weights frozen. Extensive experiments demonstrate that MSLoRA consistently improves transfer performance on classification, detection, and segmentation tasks with roughly less than 5\% of backbone parameters. The design further enables stable optimization, fast convergence, and strong cross-architecture generalization. By reweighting rather than re-tuning, MSLoRA provides a simple and universal approach for efficient adaptation of frozen vision backbones.