Approaching Dialogue State Tracking via Aligning Speech Encoders and LLMs

TL;DR

This paper presents a method for spoken Dialogue State Tracking that aligns speech encoders with large language models using a connector module, achieving state-of-the-art results on the SpokenWOZ dataset.

Contribution

The work introduces a novel approach to dialogue state tracking by bridging speech encoders and LLMs with a connector, utilizing open-source components and data augmentation.

Findings

Achieved 42.17% JGA on SpokenWOZ test set with the best model.

Fuzzy matching-based post-processing improves named entity recognition.

Fine-tuning strategies significantly impact system performance.

Abstract

In this work, we approach spoken Dialogue State Tracking (DST) by bridging the representation spaces of speech encoders and LLMs via a small connector module, with a focus on fully open-sourced and open-data components (WavLM-large, OLMo). We focus on ablating different aspects of such systems including full/LoRA adapter fine-tuning, the effect of agent turns in the dialogue history, as well as fuzzy matching-based output post-processing, which greatly improves performance of our systems on named entities in the dialogue slot values. We conduct our experiments on the SpokenWOZ dataset, and additionally utilize the Speech-Aware MultiWOZ dataset to augment our training data. Ultimately, our best-performing WavLM + connector + OLMo-1B aligned models achieve state of the art on the SpokenWOZ test set (34.66% JGA), and our system with Gemma-2-9B-instruct further surpasses this result, reaching 42.17% JGA on SpokenWOZ test.