Schema-Guided Dialogue State Tracking Task at DSTC8

TL;DR

This paper presents the Schema-Guided Dialogue State Tracking task at DSTC8, focusing on large-scale, data-efficient, and zero-shot generalizable models for virtual assistants, supported by a new dataset and baseline model.

Contribution

It introduces a new large-scale dataset, defines a challenging task for zero-shot API generalization, and provides baseline models and evaluation methods for future research.

Findings

Participants developed neural models exceeding baseline performance

Use of pre-trained encoders and data augmentation improved results

High participation indicates strong interest and progress in the field

Abstract

This paper gives an overview of the Schema-Guided Dialogue State Tracking task of the 8th Dialogue System Technology Challenge. The goal of this task is to develop dialogue state tracking models suitable for large-scale virtual assistants, with a focus on data-efficient joint modeling across domains and zero-shot generalization to new APIs. This task provided a new dataset consisting of over 16000 dialogues in the training set spanning 16 domains to highlight these challenges, and a baseline model capable of zero-shot generalization to new APIs. Twenty-five teams participated, developing a range of neural network models, exceeding the performance of the baseline model by a very high margin. The submissions incorporated a variety of pre-trained encoders and data augmentation techniques. This paper describes the task definition, dataset and evaluation methodology. We also summarize the approach and results of the submitted systems to highlight the overall trends in the state-of-the-art.