TL;DR
LiLAS 2021 introduces a user-centric living lab environment for academic search, enabling real-world evaluation of retrieval approaches through innovative infrastructure and tasks in life sciences and social sciences.
Contribution
This paper presents the LiLAS 2021 framework, including the new STELLA infrastructure and real-world evaluation tasks for academic search systems.
Findings
Successful integration of lab and real-world evaluations
Effective use of Docker containers for real-time testing
Meta-analysis of different retrieval approaches
Abstract
The Living Labs for Academic Search (LiLAS) lab aims to strengthen the concept of user-centric living labs for academic search. The methodological gap between real-world and lab-based evaluation should be bridged by allowing lab participants to evaluate their retrieval approaches in two real-world academic search systems from life sciences and social sciences. This overview paper outlines the two academic search systems LIVIVO and GESIS Search, and their corresponding tasks within LiLAS, which are ad-hoc retrieval and dataset recommendation. The lab is based on a new evaluation infrastructure named STELLA that allows participants to submit results corresponding to their experimental systems in the form of pre-computed runs and Docker containers that can be integrated into production systems and generate experimental results in real-time. Both submission types are interleaved with the…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
