Adaptor: Objective-Centric Adaptation Framework for Language Models

TL;DR

Adaptor is a framework that shifts the focus from model-centric to objective-centric training in NLP, enabling flexible experimentation with objectives for multitask learning, domain adaptation, and custom training strategies.

Contribution

It introduces the Adaptor library that facilitates objective-centric training, enhancing reproducibility and flexibility in NLP research and applications.

Findings

Effective in unsupervised domain adaptation scenarios

Supports multitask training and custom objectives

Eases reproducibility of NLP experiments

Abstract

Progress in natural language processing research is catalyzed by the possibilities given by the widespread software frameworks. This paper introduces Adaptor library that transposes the traditional model-centric approach composed of pre-training + fine-tuning steps to objective-centric approach, composing the training process by applications of selected objectives. We survey research directions that can benefit from enhanced objective-centric experimentation in multitask training, custom objectives development, dynamic training curricula, or domain adaptation. Adaptor aims to ease reproducibility of these research directions in practice. Finally, we demonstrate the practical applicability of Adaptor in selected unsupervised domain adaptation scenarios.