A Comprehensive Analysis of Adapter Efficiency

TL;DR

This paper critically evaluates adapters as a parameter-efficient fine-tuning method for NLP, finding that they do not offer significant efficiency or maintainability benefits over full fine-tuning in most NLU tasks.

Contribution

It provides a comprehensive experimental analysis showing that adapters are less efficient and less maintainable than full fine-tuning for NLU tasks, challenging their perceived advantages.

Findings

Adapters are more expensive to train than full fine-tuning.

Adapters have slightly higher deployment latency.

Full fine-tuning and multi-task training are more efficient for NLU tasks.

Abstract

Adapters have been positioned as a parameter-efficient fine-tuning (PEFT) approach, whereby a minimal number of parameters are added to the model and fine-tuned. However, adapters have not been sufficiently analyzed to understand if PEFT translates to benefits in training/deployment efficiency and maintainability/extensibility. Through extensive experiments on many adapters, tasks, and languages in supervised and cross-lingual zero-shot settings, we clearly show that for Natural Language Understanding (NLU) tasks, the parameter efficiency in adapters does not translate to efficiency gains compared to full fine-tuning of models. More precisely, adapters are relatively expensive to train and have slightly higher deployment latency. Furthermore, the maintainability/extensibility benefits of adapters can be achieved with simpler approaches like multi-task training via full fine-tuning, which also provide relatively faster training times. We, therefore, recommend that for moderately sized models for NLU tasks, practitioners should rely on full fine-tuning or multi-task training rather than using adapters. Our code is available at https://github.com/AI4Bharat/adapter-efficiency.