Scaled Prompt-Tuning for Few-Shot Natural Language Generation

TL;DR

This paper introduces Scaled Prompt-Tuning (SPT), a parameter-efficient method for few-shot natural language generation that improves performance and transferability without significantly increasing training costs.

Contribution

The paper proposes SPT, a novel PEFT approach that outperforms traditional prompt-tuning and demonstrates superior transferability in few-shot NLG tasks.

Findings

SPT surpasses conventional prompt-tuning in performance and generalization.

Intermediate SPT shows strong transferability in data-limited scenarios.

Some existing PEFT methods like Prefix-Tuning struggle on challenging few-shot NLG datasets.

Abstract

The increasingly Large Language Models (LLMs) demonstrate stronger language understanding and generation capabilities, while the memory demand and computation cost of fine-tuning LLMs on downstream tasks are non-negligible. Besides, fine-tuning generally requires a certain amount of data from individual tasks whilst data collection cost is another issue to consider in real-world applications. In this work, we focus on Parameter-Efficient Fine-Tuning (PEFT) methods for few-shot Natural Language Generation (NLG), which freeze most parameters in LLMs and tune a small subset of parameters in few-shot cases so that memory footprint, training cost, and labeling cost are reduced while maintaining or even improving the performance. We propose a Scaled Prompt-Tuning (SPT) method which surpasses conventional PT with better performance and generalization ability but without an obvious increase in training cost. Further study on intermediate SPT suggests the superior transferability of SPT in few-shot scenarios, providing a recipe for data-deficient and computation-limited circumstances. Moreover, a comprehensive comparison of existing PEFT methods reveals that certain approaches exhibiting decent performance with modest training cost such as Prefix-Tuning in prior study could struggle in few-shot NLG tasks, especially on challenging datasets.