From Little Things Big Things Grow: A Collection with Seed Studies for Medical Systematic Review Literature Search

TL;DR

This paper introduces a new test collection of real seed studies used in medical systematic review query formulation, highlighting their importance over pseudo seed studies and evaluating seed-study based retrieval methods.

Contribution

It provides a novel test collection with real seed studies and analyzes their impact on retrieval effectiveness compared to pseudo seed studies.

Findings

Real seed studies differ significantly from pseudo seed studies in representing expert queries.

Seed-study based methods improve retrieval performance over traditional approaches.

The collection enables better evaluation of seed study-guided retrieval techniques.

Abstract

Medical systematic review query formulation is a highly complex task done by trained information specialists. Complexity comes from the reliance on lengthy Boolean queries, which express a detailed research question. To aid query formulation, information specialists use a set of exemplar documents, called `seed studies', prior to query formulation. Seed studies help verify the effectiveness of a query prior to the full assessment of retrieved studies. Beyond this use of seeds, specific IR methods can exploit seed studies for guiding both automatic query formulation and new retrieval models. One major limitation of work to date is that these methods exploit `pseudo seed studies' through retrospective use of included studies (i.e., relevance assessments). However, we show pseudo seed studies are not representative of real seed studies used by information specialists. Hence, we provide a test collection with real world seed studies used to assist with the formulation of queries. To support our collection, we provide an analysis, previously not possible, on how seed studies impact retrieval and perform several experiments using seed-study based methods to compare the effectiveness of using seed studies versus pseudo seed studies. We make our test collection and the results of all of our experiments and analysis available at http://github.com/ielab/sysrev-seed-collection