Online Evaluations for Everyone: Mr. DLib's Living Lab for Scholarly Recommendations

TL;DR

This paper presents Mr. DLib's living lab, an innovative online platform enabling real-time evaluation of scholarly recommender systems through API integrations with research tools, facilitating continuous improvement and benchmarking.

Contribution

It introduces the first living lab for scholarly recommendations, providing a practical infrastructure for online evaluation and comparison of recommender algorithms in real-world settings.

Findings

Delivered 1,826,643 recommendations in 16 months

Achieved an average click-through rate of 0.21%

Supported integration with reference management software and digital libraries

Abstract

We introduce the first 'living lab' for scholarly recommender systems. This lab allows recommender-system researchers to conduct online evaluations of their novel algorithms for scholarly recommendations, i.e., recommendations for research papers, citations, conferences, research grants, etc. Recommendations are delivered through the living lab's API to platforms such as reference management software and digital libraries. The living lab is built on top of the recommender-system as-a-service Mr. DLib. Current partners are the reference management software JabRef and the CORE research team. We present the architecture of Mr. DLib's living lab as well as usage statistics on the first sixteen months of operating it. During this time, 1,826,643 recommendations were delivered with an average click-through rate of 0.21%.