Parameter-Efficient Low-Resource Dialogue State Tracking by Prompt Tuning

TL;DR

This paper introduces a prompt tuning approach for dialogue state tracking that significantly reduces parameter requirements and improves performance in low-resource settings by using soft prompt token embeddings without fine-tuning the entire language model.

Contribution

It proposes a novel soft prompt tuning method for DST that minimizes parameter tuning and enhances low-resource domain adaptation.

Findings

Reduces parameters to less than 0.5% of traditional fine-tuning methods.

Achieves better performance in low-resource DST scenarios.

Requires no tuning of the entire language model.

Abstract

Dialogue state tracking (DST) is an important step in dialogue management to keep track of users' beliefs. Existing works fine-tune all language model (LM) parameters to tackle the DST task, which requires significant data and computing resources for training and hosting. The cost grows exponentially in the real-world deployment where dozens of fine-tuned LM are used for different domains and tasks. To reduce parameter size and better utilize cross-task shared information, we propose to use soft prompt token embeddings to learn task properties. Without tuning LM parameters, our method drastically reduces the number of parameters needed to less than 0.5% of prior works while achieves better low-resource DST performance.