Living Lab Evaluation for Life and Social Sciences Search Platforms -- LiLAS at CLEF 2021
Philipp Schaer, Johann Schaible, Leyla Jael Castro

TL;DR
This paper discusses the LiLAS living lab initiative at CLEF 2021, which promotes user-centric evaluation of academic search systems in real-world settings for life and social sciences, emphasizing the importance of multilayered relevance.
Contribution
It introduces a novel living lab framework for academic search evaluation that integrates real-world systems and enables comparison of retrieval approaches in user-centric scenarios.
Findings
Participants successfully integrated their retrieval approaches into real-world systems.
The infrastructure facilitated comparison of different approaches in real-world settings.
The study highlights the importance of user-centric evaluation in academic search.
Abstract
Meta-evaluation studies of system performances in controlled offline evaluation campaigns, like TREC and CLEF, show a need for innovation in evaluating IR-systems. The field of academic search is no exception to this. This might be related to the fact that relevance in academic search is multilayered and therefore the aspect of user-centric evaluation is becoming more and more important. The Living Labs for Academic Search (LiLAS) lab aims to strengthen the concept of user-centric living labs for the domain of academic search by allowing participants to evaluate their retrieval approaches in two real-world academic search systems from the life sciences and the social sciences. To this end, we provide participants with metadata on the systems' content as well as candidate lists with the task to rank the most relevant candidate to the top. Using the STELLA-infrastructure, we allow…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
