Evaluation of Coordination Techniques in Synchronous Collaborative Information Retrieval

TL;DR

This paper proposes an evaluation methodology for Synchronous Collaborative Information Retrieval (SCIR), enabling effective assessment of coordination techniques through user simulation, and presents empirical studies on division of labour and knowledge sharing.

Contribution

It introduces a reusable evaluation framework for SCIR systems and applies it to analyze different coordination techniques in collaborative search.

Findings

Division of labour impacts search efficiency.

Sharing knowledge improves retrieval effectiveness.

Evaluation methodology is adaptable for various SCIR scenarios.

Abstract

Traditional Information Retrieval (IR) research has focussed on a single user interaction modality, where a user searches to satisfy an information need. Recent advances in web technologies and computer hardware have enabled multiple users to collaborate on many computer-supported tasks, therefore there is an increasing opportunity to support two or more users searching together at the same time in order to satisfy a shared information need, which we refer to as Synchronous Collaborative Information Retrieval (SCIR). SCIR systems represent a significant paradigmatic shift from traditional IR systems. In order to support effective SCIR, new techniques are required to coordinate users' activities. In addition, the novel domain of SCIR presents challenges for effective evaluations of these systems. In this paper we will propose an effective and re-usable evaluation methodology based on simulating users searching together. We will outline how we have used this evaluation in empirical studies of the effects of different division of labour and sharing of knowledge techniques for SCIR.