Domain Expansion: Parameter-Efficient Modules as Building Blocks for Composite Domains

TL;DR

This paper introduces a method for composing parameter-efficient modules trained on individual domains to represent complex composite domains, demonstrated through personality trait modeling without additional fine-tuning.

Contribution

It proposes a novel approach to compose PEMs in the weight space for domain generalization, enabling efficient modeling of composite domains without extra fine-tuning.

Findings

PEFT effectively fine-tunes PEMs for individual domains.

Composed PEMs accurately represent composite personalities.

The method achieves high validation scores in personality prediction.

Abstract

Parameter-Efficient Fine-Tuning (PEFT) is an efficient alternative to full scale fine-tuning, gaining popularity recently. With pre-trained model sizes growing exponentially, PEFT can be effectively utilized to fine-tune compact modules, Parameter-Efficient Modules (PEMs), trained to be domain experts over diverse domains. In this project, we explore composing such individually fine-tuned PEMs for distribution generalization over the composite domain. To compose PEMs, simple composing functions are used that operate purely on the weight space of the individually fine-tuned PEMs, without requiring any additional fine-tuning. The proposed method is applied to the task of representing the 16 Myers-Briggs Type Indicator (MBTI) composite personalities via 4 building block dichotomies, comprising of 8 individual traits which can be merged (composed) to yield a unique personality. We evaluate the individual trait PEMs and the composed personality PEMs via an online MBTI personality quiz questionnaire, validating the efficacy of PEFT to fine-tune PEMs and merging PEMs without further fine-tuning for domain composition.