Unfreeze with Care: Space-Efficient Fine-Tuning of Semantic Parsing Models

TL;DR

This paper explores efficient methods for adapting large pretrained language models to semantic parsing tasks, comparing prefix tuning and bias-term tuning, and proposing modifications to improve performance while maintaining parameter efficiency.

Contribution

It introduces a modified prefix tuning approach with special token embeddings that enhances semantic parsing performance without losing parameter savings.

Findings

Bias-term tuning performs well on semantic parsing.

Modified prefix tuning with special tokens achieves strong results.

Parameter-efficient tuning methods can match full fine-tuning performance.

Abstract

Semantic parsing is a key NLP task that maps natural language to structured meaning representations. As in many other NLP tasks, SOTA performance in semantic parsing is now attained by fine-tuning a large pretrained language model (PLM). While effective, this approach is inefficient in the presence of multiple downstream tasks, as a new set of values for all parameters of the PLM needs to be stored for each task separately. Recent work has explored methods for adapting PLMs to downstream tasks while keeping most (or all) of their parameters frozen. We examine two such promising techniques, prefix tuning and bias-term tuning, specifically on semantic parsing. We compare them against each other on two different semantic parsing datasets, and we also compare them against full and partial fine-tuning, both in few-shot and conventional data settings. While prefix tuning is shown to do poorly for semantic parsing tasks off the shelf, we modify it by adding special token embeddings, which results in very strong performance without compromising parameter savings.