TREC CAsT 2019: The Conversational Assistance Track Overview

TL;DR

The TREC CAsT 2019 track introduced a large-scale conversational search test collection, evaluated diverse methods including neural models, and highlighted the performance gap between automatic and manual query reformulations.

Contribution

This paper provides an overview of the first TREC CAsT track, including dataset creation, participant methods, and analysis of results for conversational information seeking systems.

Findings

Neural reranking with BERT was a common effective approach.

Manual query rewrites significantly outperformed automatic systems.

Diverse methods including generative models were explored for conversational search.

Abstract

The Conversational Assistance Track (CAsT) is a new track for TREC 2019 to facilitate Conversational Information Seeking (CIS) research and to create a large-scale reusable test collection for conversational search systems. The document corpus is 38,426,252 passages from the TREC Complex Answer Retrieval (CAR) and Microsoft MAchine Reading COmprehension (MARCO) datasets. Eighty information seeking dialogues (30 train, 50 test) are an average of 9 to 10 questions long. Relevance assessments are provided for 30 training topics and 20 test topics. This year 21 groups submitted a total of 65 runs using varying methods for conversational query understanding and ranking. Methods include traditional retrieval based methods, feature based learning-to-rank, neural models, and knowledge enhanced methods. A common theme through the runs is the use of BERT-based neural reranking methods. Leading methods also employed document expansion, conversational query expansion, and generative language models for conversational query rewriting (GPT-2). The results show a gap between automatic systems and those using the manually resolved utterances, with a 35% relative improvement of manual rewrites over the best automatic system.