Overview of LiLAS 2021 -- Living Labs for Academic Search

TL;DR

LiLAS 2021 introduces a user-centric living lab environment for academic search, enabling real-world evaluation of retrieval approaches through innovative infrastructure and tasks in life sciences and social sciences.

Contribution

This paper presents the LiLAS 2021 framework, including the new STELLA infrastructure and real-world evaluation tasks for academic search systems.

Findings

Successful integration of lab and real-world evaluations

Effective use of Docker containers for real-time testing

Meta-analysis of different retrieval approaches

Abstract

The Living Labs for Academic Search (LiLAS) lab aims to strengthen the concept of user-centric living labs for academic search. The methodological gap between real-world and lab-based evaluation should be bridged by allowing lab participants to evaluate their retrieval approaches in two real-world academic search systems from life sciences and social sciences. This overview paper outlines the two academic search systems LIVIVO and GESIS Search, and their corresponding tasks within LiLAS, which are ad-hoc retrieval and dataset recommendation. The lab is based on a new evaluation infrastructure named STELLA that allows participants to submit results corresponding to their experimental systems in the form of pre-computed runs and Docker containers that can be integrated into production systems and generate experimental results in real-time. Both submission types are interleaved with the results provided by the productive systems allowing for a seamless presentation and evaluation. The evaluation of results and a meta-analysis of the different tasks and submission types complement this overview.