Empowering parameter-efficient transfer learning by recognizing the kernel structure in self-attention

TL;DR

This paper introduces kernel-wise adapters for parameter-efficient transfer learning in language models, leveraging the kernel structure of self-attention to improve task performance with fewer tunable parameters.

Contribution

It proposes kernel-wise adapters that utilize the kernel structure in self-attention, enabling more effective and efficient parameter tuning for transfer learning.

Findings

Achieves comparable or better performance than existing methods.

Enables separate tuning for each attention head.

Demonstrates effectiveness across diverse NLP tasks.

Abstract

The massive amount of trainable parameters in the pre-trained language models (PLMs) makes them hard to be deployed to multiple downstream tasks. To address this issue, parameter-efficient transfer learning methods have been proposed to tune only a few parameters during fine-tuning while freezing the rest. This paper looks at existing methods along this line through the \textit{kernel lens}. Motivated by the connection between self-attention in transformer-based PLMs and kernel learning, we propose \textit{kernel-wise adapters}, namely \textit{Kernel-mix}, that utilize the kernel structure in self-attention to guide the assignment of the tunable parameters. These adapters use guidelines found in classical kernel learning and enable separate parameter tuning for each attention head. Our empirical results, over a diverse set of natural language generation and understanding tasks, show that our proposed adapters can attain or improve the strong performance of existing baselines.