The Task-oriented Queries Benchmark (ToQB)

TL;DR

The paper introduces ToQB, a new benchmark for task-oriented queries, created using an innovative methodology that leverages existing dialogue datasets and LLMs, to evaluate virtual assistants and chatbots.

Contribution

It presents a novel, automated approach for generating a comprehensive task-oriented queries benchmark from dialogue datasets using LLMs, filling a gap in NLP evaluation tools.

Findings

Successfully generated ToQB dataset across three domains.

Demonstrated customization of LLM prompts for different domains.

Provided a framework for community-driven expansion of ToQB.

Abstract

Task-oriented queries (e.g., one-shot queries to play videos, order food, or call a taxi) are crucial for assessing the quality of virtual assistants, chatbots, and other large language model (LLM)-based services. However, a standard benchmark for task-oriented queries is not yet available, as existing benchmarks in the relevant NLP (Natural Language Processing) fields have primarily focused on task-oriented dialogues. Thus, we present a new methodology for efficiently generating the Task-oriented Queries Benchmark (ToQB) using existing task-oriented dialogue datasets and an LLM service. Our methodology involves formulating the underlying NLP task to summarize the original intent of a speaker in each dialogue, detailing the key steps to perform the devised NLP task using an LLM service, and outlining a framework for automating a major part of the benchmark generation process. Through a case study encompassing three domains (i.e., two single-task domains and one multi-task domain), we demonstrate how to customize the LLM prompts (e.g., omitting system utterances or speaker labels) for those three domains and characterize the generated task-oriented queries. The generated ToQB dataset is made available to the public. We further discuss new domains that can be added to ToQB by community contributors and its practical applications.