From Baseline to Top Performer: A Reproducibility Study of Approaches at the TREC 2021 Conversational Assistance Track

TL;DR

This study reproduces and analyzes the effectiveness of TREC 2021 Conversational Assistance systems, revealing reproducibility challenges, and demonstrates how advanced retrieval techniques and dataset choices impact system performance.

Contribution

It provides a detailed reproducibility analysis of top TREC systems, identifies missing practical information, and explores the effects of different pipeline components on performance.

Findings

Reproducibility within 19% margin of original results

The performance gap between baseline and top system shrinks from 18% to 5%

Advanced retrieval techniques improve system effectiveness

Abstract

This paper reports on an effort of reproducing the organizers' baseline as well as the top performing participant submission at the 2021 edition of the TREC Conversational Assistance track. TREC systems are commonly regarded as reference points for effectiveness comparison. Yet, the papers accompanying them have less strict requirements than peer-reviewed publications, which can make reproducibility challenging. Our results indicate that key practical information is indeed missing. While the results can be reproduced within a 19% relative margin with respect to the main evaluation measure, the relative difference between the baseline and the top performing approach shrinks from the reported 18% to 5%. Additionally, we report on a new set of experiments aimed at understanding the impact of various pipeline components. We show that end-to-end system performance can indeed benefit from advanced retrieval techniques in either stage of a two-stage retrieval pipeline. We also measure the impact of the dataset used for fine-tuning the query rewriter and find that employing different query rewriting methods in different stages of the retrieval pipeline might be beneficial. Moreover, these results are shown to generalize across the 2020 and 2021 editions of the track. We conclude our study with a list of lessons learned and practical suggestions.