Hyperdecoders: Instance-specific decoders for multi-task NLP

TL;DR

Hyperdecoders introduce input-conditioned hypernetworks that generate instance-specific decoder adaptations in NLP, enhancing flexibility and efficiency over traditional methods, and outperforming some full fine-tuning approaches.

Contribution

This work presents a novel hypernetwork-based method for generating input-specific decoders in NLP, improving parameter efficiency and task performance.

Findings

Outperforms previous parameter-efficient fine-tuning methods.

Often surpasses full model fine-tuning in accuracy.

Embeddings are sensitive to output label and type.

Abstract

We investigate input-conditioned hypernetworks for multi-tasking in NLP, generating parameter-efficient adaptations for a decoder using a hypernetwork conditioned on the output of an encoder. This approach produces a unique decoder adaptation for every input instance, allowing the network a larger degree of flexibility than prior work that only produces one decoder adaptation per task. We apply our method to sequence classification tasks, extractive QA, and summarisation and find that it surpasses previous parameter efficient fine-tuning methods and often outperforms fully finetuning the underlying model. An analysis of the embeddings used by our hypernetwork shows that they are sensitive to output label and type, suggesting that our approach better maps from encoder representations to output labels. Our code is publicly available at https://github.com/allenai/hyperdecoders.