Efficient Attribute Injection for Pretrained Language Models

TL;DR

This paper introduces a lightweight, memory-efficient attribute injection method for pretrained language models, enhancing their performance across diverse NLP tasks by extending adapters with low-rank and hypercomplex techniques.

Contribution

It proposes a novel attribute injection approach using adapters with low-rank and hypercomplex methods, improving efficiency and effectiveness over prior techniques.

Findings

Outperforms previous attribute injection methods

Achieves state-of-the-art results on multiple datasets

Reduces parameter increase with low-rank and hypercomplex approximations

Abstract

Metadata attributes (e.g., user and product IDs from reviews) can be incorporated as additional inputs to neural-based NLP models, by modifying the architecture of the models, in order to improve their performance. Recent models however rely on pretrained language models (PLMs), where previously used techniques for attribute injection are either nontrivial or ineffective. In this paper, we propose a lightweight and memory-efficient method to inject attributes to PLMs. We extend adapters, i.e. tiny plug-in feed-forward modules, to include attributes both independently of or jointly with the text. To limit the increase of parameters especially when the attribute vocabulary is large, we use low-rank approximations and hypercomplex multiplications, significantly decreasing the total parameters. We also introduce training mechanisms to handle domains in which attributes can be multi-labeled or sparse. Extensive experiments and analyses on eight datasets from different domains show that our method outperforms previous attribute injection methods and achieves state-of-the-art performance on various datasets.