The Power of Prompt Tuning for Low-Resource Semantic Parsing

TL;DR

This paper demonstrates that prompt tuning significantly improves low-resource semantic parsing performance over traditional fine-tuning methods, especially with larger models, by effectively leveraging pre-trained language models.

Contribution

It introduces prompt tuning for semantic parsing and shows its superiority over fine-tuning and baseline models on low-resource datasets.

Findings

Prompt tuning outperforms fine-tuning on low-resource datasets.

Larger models benefit more from prompt tuning in semantic parsing.

Prompt tuned T5 models generate more diverse representations with increased scale.

Abstract

Prompt tuning has recently emerged as an effective method for adapting pre-trained language models to a number of language understanding and generation tasks. In this paper, we investigate prompt tuning for semantic parsing -- the task of mapping natural language utterances onto formal meaning representations. On the low-resource splits of Overnight and TOPv2, we find that a prompt tuned T5-xl significantly outperforms its fine-tuned counterpart, as well as strong GPT-3 and BART baselines. We also conduct ablation studies across different model scales and target representations, finding that, with increasing model scale, prompt tuned T5 models improve at generating target representations that are far from the pre-training distribution.