Zero-Shot Cross-Domain Dialogue State Tracking via Dual Low-Rank Adaptation

TL;DR

This paper introduces DualLoRA, a novel low-rank adaptation method that enhances zero-shot dialogue state tracking across domains by effectively integrating prompts into transformer models without increasing inference latency.

Contribution

The paper proposes DualLoRA, a dual low-rank adaptation architecture that improves prompt influence throughout transformer layers for zero-shot DST, outperforming existing methods.

Findings

Outperforms baseline methods on MultiWOZ and SGD datasets

Enhances prompt influence across all transformer layers

Maintains inference efficiency without latency increase

Abstract

Zero-shot dialogue state tracking (DST) seeks to enable dialogue systems to transition to unfamiliar domains without manual annotation or extensive retraining. Prior research has approached this objective by embedding prompts into language models (LMs). Common methodologies include integrating prompts at the input layer or introducing learnable variables at each transformer layer. Nonetheless, each strategy exhibits inherent limitations. Prompts integrated at the input layer risk underutilization, with their impact potentially diminishing across successive transformer layers. Conversely, the addition of learnable variables to each layer can complicate the training process and increase inference latency. To tackle the issues mentioned above, this paper proposes Dual Low-Rank Adaptation (DualLoRA), a plug-and-play architecture designed for zero-shot DST. DualLoRA incorporates two distinct Low-Rank Adaptation (LoRA) components, targeting both dialogue context processing and prompt optimization, to ensure the comprehensive influence of prompts throughout the transformer model layers. This is achieved without incurring additional inference latency, showcasing an efficient integration into existing architectures. Through rigorous evaluation on the MultiWOZ and SGD datasets, DualLoRA demonstrates notable improvements across multiple domains, outperforming traditional baseline methods in zero-shot settings. Our code is accessible at: \url{https://github.com/suntea233/DualLoRA}.