PARA: Parameter-Efficient Fine-tuning with Prompt Aware Representation Adjustment

TL;DR

PARA is a new parameter-efficient fine-tuning method that uses prompt-aware representation adjustment, outperforming existing methods like LoRA in efficiency and performance for multi-tenant transformer applications.

Contribution

Introduces PARA, a lightweight PEFT technique with prompt-aware vector generators, improving efficiency and performance in single-backbone multi-tenant scenarios.

Findings

Surpasses current PEFT benchmarks in performance.

More efficient than LoRA in multi-tenant settings.

Effective across diverse tasks.

Abstract

In the realm of parameter-efficient fine-tuning (PEFT) methods, while options like LoRA are available, there is a persistent demand in the industry for a PEFT approach that excels in both efficiency and performance within the context of single-backbone multi-tenant applications. This paper introduces a new and straightforward PEFT technique, termed \underline{P}rompt \underline{A}ware \underline{R}epresentation \underline{A}djustment (PARA). The core of our proposal is to integrate a lightweight vector generator within each Transformer layer. This generator produces vectors that are responsive to input prompts, thereby adjusting the hidden representations accordingly. Our extensive experimentation across diverse tasks has yielded promising results. Firstly, the PARA method has been shown to surpass current PEFT benchmarks in terms of performance, despite having a similar number of adjustable parameters. Secondly, it has proven to be more efficient than LoRA in the single-backbone multi-tenant scenario, highlighting its significant potential for industrial adoption.