Dialogue for Prompting: a Policy-Gradient-Based Discrete Prompt Generation for Few-shot Learning

TL;DR

This paper introduces a novel reinforcement learning approach, DP2O, for discrete prompt optimization in few-shot NLP tasks, leveraging dialogue strategies and efficient metrics to outperform existing methods with less computational cost.

Contribution

The paper proposes a new RL-based discrete prompt optimization method, DP2O, using dialogue alignment and a prompt screening metric, improving efficiency and performance over prior approaches.

Findings

DP2O outperforms SOTA by 1.52% in accuracy on four datasets.

DP2O requires only 0.67% of PLM parameters for training.

DP2O demonstrates strong universality, robustness, and generalization.

Abstract

Prompt-based pre-trained language models (PLMs) paradigm have succeeded substantially in few-shot natural language processing (NLP) tasks. However, prior discrete prompt optimization methods require expert knowledge to design the base prompt set and identify high-quality prompts, which is costly, inefficient, and subjective. Meanwhile, existing continuous prompt optimization methods improve the performance by learning the ideal prompts through the gradient information of PLMs, whose high computational cost, and low readability and generalizability are often concerning. To address the research gap, we propose a Dialogue-comprised Policy-gradient-based Discrete Prompt Optimization ($DP_2O$) method. We first design a multi-round dialogue alignment strategy for readability prompt set generation based on GPT-4. Furthermore, we propose an efficient prompt screening metric to identify high-quality prompts with linear complexity. Finally, we construct a reinforcement learning (RL) framework based on policy gradients to match the prompts to inputs optimally. By training a policy network with only 0.67% of the PLM parameter size on the tasks in the few-shot setting, $DP_2O$ outperforms the state-of-the-art (SOTA) method by 1.52% in accuracy on average on four open-source datasets. Moreover, subsequent experiments also demonstrate that $DP_2O$ has good universality, robustness, and generalization ability.