Hadamard Adapter: An Extreme Parameter-Efficient Adapter Tuning Method for Pre-trained Language Models

TL;DR

This paper introduces the Hadamard adapter, a highly parameter-efficient method for fine-tuning pre-trained language models by applying element-wise linear transformations on self-attention outputs, achieving competitive performance with minimal parameters.

Contribution

The paper proposes the Hadamard adapter, the most parameter-efficient adapter to date, and explores shared tuning patterns for further reduction in downstream tasks.

Findings

Achieves competitive performance with only 0.033% of parameters of full fine-tuning.

Requires fewer parameters than existing adapters.

Redundant layers can be removed for even greater efficiency, reducing parameters to 0.022%.

Abstract

Recent years, Pre-trained Language models (PLMs) have swept into various fields of artificial intelligence and achieved great success. However, most PLMs, such as T5 and GPT3, have a huge amount of parameters, fine-tuning them is often expensive and time consuming, and storing them takes up a lot of space. Therefore, it is necessary to adopt a parameter-efficient approach to reduce parameters of PLMs in fine-tuning without compromising their performance in downstream tasks. In this paper, we design a novel adapter which only acts on self-attention outputs in PLMs. This adapter adopts element-wise linear transformation using Hadamard product, hence named as Hadamard adapter, requires the fewest parameters compared to previous parameter-efficient adapters. In addition, we also summarize some tuning patterns for Hadamard adapter shared by various downstream tasks, expecting to provide some guidance for further parameter reduction with shared adapters in future studies. The experiments conducted on the widely-used GLUE benchmark with several SOTA PLMs prove that the Hadamard adapter achieves competitive performance with only 0.033\% parameters compared with full fine-tuning, and it has the fewest parameters compared with other adapters. Moreover, we further find that there is also some redundant layers in the Hadamard adapter which can be removed to achieve more parameter efficiency with only 0.022\% parameters.