Diving into Kronecker Adapters: Component Design Matters

TL;DR

This paper investigates how the design of Kronecker adapters influences their capacity and performance, proposing a new component design method that improves fine-tuning of large models.

Contribution

It introduces Component Designed Kronecker Adapters (CDKA), analyzing component configurations and providing practical guidelines for better adapter design.

Findings

Component configuration significantly affects adapter capacity.

CDKA outperforms existing Kronecker adapters in NLP tasks.

Guidelines improve adapter efficiency and stability.

Abstract

Kronecker adapters have emerged as a promising approach for fine-tuning large-scale models, enabling high-rank updates through tunable component structures. However, existing work largely treats the component structure as a fixed or heuristic design choice, leaving the dimensions and number of Kronecker components underexplored. In this paper, we identify component structure as a key factor governing the capacity of Kronecker adapters. We perform a fine-grained analysis of both the dimensions and number of Kronecker components. In particular, we show that the alignment between Kronecker adapters and full fine-tuning depends on component configurations. Guided by these insights, we propose Component Designed Kronecker Adapters (CDKA). We further provide parameter-budget-aware configuration guidelines and a tailored training stabilization strategy for practical deployment. Experiments across various natural language processing tasks demonstrate the effectiveness of CDKA. Code is available at https://github.com/rainstonee/CDKA.