Benchmarking Clinical Decision Support Search

TL;DR

This paper presents a benchmarking framework for clinical decision support search systems, enabling comparison of different retrieval techniques and facilitating reproducible research in evidence-based medicine.

Contribution

It introduces a stable platform for benchmarking clinical search methods, allowing systematic comparison and statistical testing of various document and query processing techniques.

Findings

Different methods show varied effectiveness

Benchmarking enables reproducibility and comparison

Statistical analysis supports hypothesis testing

Abstract

Finding relevant literature underpins the practice of evidence-based medicine. From 2014 to 2016, TREC conducted a clinical decision support track, wherein participants were tasked with finding articles relevant to clinical questions posed by physicians. In total, 87 teams have participated over the past three years, generating 395 runs. During this period, each team has trialled a variety of methods. While there was significant overlap in the methods employed by different teams, the results were varied. Due to the diversity of the platforms used, the results arising from the different techniques are not directly comparable, reducing the ability to build on previous work. By using a stable platform, we have been able to compare different document and query processing techniques, allowing us to experiment with different search parameters. We have used our system to reproduce leading teams runs, and compare the results obtained. By benchmarking our indexing and search techniques, we can statistically test a variety of hypotheses, paving the way for further research.