TREC iKAT 2023: A Test Collection for Evaluating Conversational and Interactive Knowledge Assistants

TL;DR

The paper introduces the TREC iKAT 2023 collection, a comprehensive benchmark dataset designed to evaluate conversational search agents in personalized, interactive contexts with diverse user personas and complex relevance assessments.

Contribution

It presents a novel test collection with personalized dialogues, PTKB integration, and multi-dimensional response assessments to advance research in conversational knowledge assistants.

Findings

Provides 36 dialogues over 20 topics with relevance assessments

Includes evaluations on response relevance, completeness, groundedness, and naturalness

Challenges CSA to handle diverse personal contexts and user personas

Abstract

Conversational information seeking has evolved rapidly in the last few years with the development of Large Language Models (LLMs), providing the basis for interpreting and responding in a naturalistic manner to user requests. The extended TREC Interactive Knowledge Assistance Track (iKAT) collection aims to enable researchers to test and evaluate their Conversational Search Agents (CSA). The collection contains a set of 36 personalized dialogues over 20 different topics each coupled with a Personal Text Knowledge Base (PTKB) that defines the bespoke user personas. A total of 344 turns with approximately 26,000 passages are provided as assessments on relevance, as well as additional assessments on generated responses over four key dimensions: relevance, completeness, groundedness, and naturalness. The collection challenges CSA to efficiently navigate diverse personal contexts, elicit pertinent persona information, and employ context for relevant conversations. The integration of a PTKB and the emphasis on decisional search tasks contribute to the uniqueness of this test collection, making it an essential benchmark for advancing research in conversational and interactive knowledge assistants.