Evaluating Research Dataset Recommendations in a Living Lab

TL;DR

This paper presents a live-evaluated system for recommending research datasets by modeling the task as a retrieval problem, combining multiple ranking methods, and demonstrating its effectiveness through real user interaction data.

Contribution

It introduces a multistage retrieval approach for research dataset recommendations and evaluates it in a live setting, outperforming other systems.

Findings

The system effectively utilizes click feedback and document embeddings.

Live evaluation shows the system outperforms competing approaches.

Pre-testing with pseudo test collections aids in fine-tuning the system.

Abstract

The search for research datasets is as important as laborious. Due to the importance of the choice of research data in further research, this decision must be made carefully. Additionally, because of the growing amounts of data in almost all areas, research data is already a central artifact in empirical sciences. Consequentially, research dataset recommendations can beneficially supplement scientific publication searches. We formulated the recommendation task as a retrieval problem by focussing on broad similarities between research datasets and scientific publications. In a multistage approach, initial recommendations were retrieved by the BM25 ranking function and dynamic queries. Subsequently, the initial ranking was re-ranked utilizing click feedback and document embeddings. The proposed system was evaluated live on real user interaction data using the STELLA infrastructure in the LiLAS Lab at CLEF 2021. Our experimental system could efficiently be fine-tuned before the live evaluation by pre-testing the system with a pseudo test collection based on prior user interaction data from the live system. The results indicate that the experimental system outperforms the other participating systems.