Output Composability of QLoRA PEFT Modules for Plug-and-Play Attribute-Controlled Text Generation

TL;DR

This paper investigates methods for combining PEFT modules, specifically QLoRA, for flexible, attribute-controlled text generation, demonstrating effective output composition techniques across multiple models and tasks.

Contribution

It introduces and evaluates three approaches for generalizing PEFT modules beyond single-task training, highlighting the effectiveness of output summation for composition.

Findings

Summing PEFT module outputs often outperforms other composition methods.

Three-module output composition improves sentiment control performance by 2% points.

Output composition maintains performance even when compared to specialized single-task modules.

Abstract

Parameter-efficient fine-tuning (PEFT) techniques offer task-specific fine-tuning at a fraction of the cost of full fine-tuning, but require separate fine-tuning for every new task (combination). In this paper, we explore three ways of generalising beyond single-task training/inference: (i) training on combinations of multiple, related datasets; (ii) at inference, composing the weight matrices of separately trained PEFT modules; and (iii) at inference, composing the outputs of separately trained PEFT modules. We test these approaches on three different LLMs, QLoRA as the PEFT technique, and three sets of controlled text generation datasets for sentiment control, topic control, and multi-attribute control. We find that summing PEFT module outputs is a particularly strong composition method, which consistently either outperforms or matches the performance of alternative approaches. This is the case even when comparing against single-task specialised modules on the single-task test set, where three-module output composition achieves an average 2% point performance increase across all models for sentiment control.