AdapterBias: Parameter-efficient Token-dependent Representation Shift for Adapters in NLP Tasks

TL;DR

AdapterBias introduces a simple, parameter-efficient adapter architecture that applies token-dependent shifts to transformer outputs, significantly reducing parameters needed for NLP tasks with minimal performance loss.

Contribution

It proposes a novel adapter design that uses token-dependent shifts, requiring fewer parameters and automatically focusing on task-relevant tokens, improving efficiency in NLP models.

Findings

Drastically reduces trainable parameters compared to previous adapters.

Maintains near-original performance levels on downstream tasks.

Automatically emphasizes task-relevant tokens in representation shifts.

Abstract

Transformer-based pre-trained models with millions of parameters require large storage. Recent approaches tackle this shortcoming by training adapters, but these approaches still require a relatively large number of parameters. In this study, AdapterBias, a surprisingly simple yet effective adapter architecture, is proposed. AdapterBias adds a token-dependent shift to the hidden output of transformer layers to adapt to downstream tasks with only a vector and a linear layer. Extensive experiments are conducted to demonstrate the effectiveness of AdapterBias. The experiments show that our proposed method can dramatically reduce the trainable parameters compared to the previous works with a minimal decrease in task performances compared with fine-tuned pre-trained models. We further find that AdapterBias automatically learns to assign more significant representation shifts to the tokens related to the task in consideration.