Controllable Conversational Theme Detection Track at DSTC 12

TL;DR

This paper introduces a new controllable theme detection task for conversational analytics, framing it as a DSTC 12 challenge that involves clustering and labeling dialog utterances with adjustable granularity based on user preferences.

Contribution

It presents a novel problem formulation for controllable conversational theme detection, along with dataset, evaluation metrics, and a public competition track to advance research in this area.

Findings

Multiple participant solutions analyzed

Controllability of theme granularity demonstrated

Open dataset and code released for future research

Abstract

Conversational analytics has been on the forefront of transformation driven by the advances in Speech and Natural Language Processing techniques. Rapid adoption of Large Language Models (LLMs) in the analytics field has taken the problems that can be automated to a new level of complexity and scale. In this paper, we introduce Theme Detection as a critical task in conversational analytics, aimed at automatically identifying and categorizing topics within conversations. This process can significantly reduce the manual effort involved in analyzing expansive dialogs, particularly in domains like customer support or sales. Unlike traditional dialog intent detection, which often relies on a fixed set of intents for downstream system logic, themes are intended as a direct, user-facing summary of the conversation's core inquiry. This distinction allows for greater flexibility in theme surface forms and user-specific customizations. We pose Controllable Conversational Theme Detection problem as a public competition track at Dialog System Technology Challenge (DSTC) 12 -- it is framed as joint clustering and theme labeling of dialog utterances, with the distinctive aspect being controllability of the resulting theme clusters' granularity achieved via the provided user preference data. We give an overview of the problem, the associated dataset and the evaluation metrics, both automatic and human. Finally, we discuss the participant teams' submissions and provide insights from those. The track materials (data and code) are openly available in the GitHub repository.