WHOSE - A Tool for Whole-Session Analysis in IIR

TL;DR

WHOSE is a versatile framework designed for comprehensive whole-session analysis in Interactive Information Retrieval, enabling researchers to evaluate and compare user search behaviors across real-world IR systems using integrated logging and interactive visualization.

Contribution

The paper introduces a novel framework for whole-session evaluation in IIR that can utilize uncontrolled real-world data and integrate seamlessly with existing IR systems.

Findings

Framework supports analysis of real user sessions across different IR systems.

Interactive GUI allows detailed exploration from overview to individual sessions.

Compatible with existing log data and adaptable to various IR environments.

Abstract

One of the main challenges in Interactive Information Retrieval (IIR) evaluation is the development and application of re-usable tools that allow researchers to analyze search behavior of real users in different environments and different domains, but with comparable results. Furthermore, IIR recently focuses more on the analysis of whole sessions, which includes all user interactions that are carried out within a session but also across several sessions by the same user. Some frameworks have already been proposed for the evaluation of controlled experiments in IIR, but yet no framework is available for interactive evaluation of search behavior from real-world information retrieval (IR) systems with real users. In this paper we present a framework for whole-session evaluation that can also utilize these uncontrolled data sets. The logging component can easily be integrated into real-world IR systems for generating and analyzing new log data. Furthermore, due to a supplementary mapping it is also possible to analyze existing log data. For every IR system different actions and filters can be defined. This allows system operators and researchers to use the framework for the analysis of user search behavior in their IR systems and to compare it with others. Using a graphical user interface they have the possibility to interactively explore the data set from a broad overview down to individual sessions.