Online Evaluations for Everyone: Mr. DLib's Living Lab for Scholarly Recommendations
Joeran Beel, Andrew Collins, Oliver Kopp, Linus W. Dietz and, Petr Knoth

TL;DR
This paper presents Mr. DLib's living lab, an innovative online platform enabling real-time evaluation of scholarly recommender systems through API integrations with research tools, facilitating continuous improvement and benchmarking.
Contribution
It introduces the first living lab for scholarly recommendations, providing a practical infrastructure for online evaluation and comparison of recommender algorithms in real-world settings.
Findings
Delivered 1,826,643 recommendations in 16 months
Achieved an average click-through rate of 0.21%
Supported integration with reference management software and digital libraries
Abstract
We introduce the first 'living lab' for scholarly recommender systems. This lab allows recommender-system researchers to conduct online evaluations of their novel algorithms for scholarly recommendations, i.e., recommendations for research papers, citations, conferences, research grants, etc. Recommendations are delivered through the living lab's API to platforms such as reference management software and digital libraries. The living lab is built on top of the recommender-system as-a-service Mr. DLib. Current partners are the reference management software JabRef and the CORE research team. We present the architecture of Mr. DLib's living lab as well as usage statistics on the first sixteen months of operating it. During this time, 1,826,643 recommendations were delivered with an average click-through rate of 0.21%.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
