Technical Report -- Competition Solution for Prompt Tuning using Pretrained Language Model

TL;DR

This paper presents a competition solution for prompt tuning of pretrained language models using derivative-free optimization in a black-box setting, incorporating multiple modifications to improve performance.

Contribution

It introduces a novel combination of techniques for prompt tuning in LMaaS scenarios, including label word selection, rolling updates, multi-task loss, and ensemble methods.

Findings

Enhanced generalization through ensemble methods

Effective prompt tuning with multiple label words

Discussion on dataset impact and competition insights

Abstract

Prompt tuning recently becomes a hot-spot in the applications of large pretrained language models on specific downstream tasks. Regarding the Language Model as a Service (LMaaS), black-box tuning using derivative-free optimization (DFO) provides a novel approach to expand the practical scenarios of pretrained models and enrich the researches of few-shot learning. In this report, we present our solution in this competition that is based on the LMaaS scenario. Our solution consists of several modifications to BBTv2, including multiple label words, selection of P0, rolling update strategy, multi-task loss from MLP classifier, and finally using the ensemble method to further improve generalization ability. We also shared some strategies that we tried but didn't use in the final submission for further discussion. In the end we raised a question about the SNLI dataset and the impact on the results, as well as our concerns about the competition.