SenseSeek Dataset: Multimodal Sensing to Study Information Seeking Behaviors

TL;DR

SenseSeek is a pioneering multimodal dataset capturing physiological and behavioral data during complex information search tasks, enabling deeper understanding of cognitive processes involved in information seeking.

Contribution

This paper introduces the first dataset combining physiological signals and interaction data during multi-stage information search tasks using consumer sensors.

Findings

Baseline analysis shows sensor data varies with cognitive intent.

Data effectively discriminates different search stages.

Physiological signals correlate with information seeking behaviors.

Abstract

Information processing tasks involve complex cognitive mechanisms that are shaped by various factors, including individual goals, prior experience, and system environments. Understanding such behaviors requires a sophisticated and personalized data capture of how one interacts with modern information systems (e.g., web search engines). Passive sensors, such as wearables, capturing physiological and behavioral data, have the potential to provide solutions in this context. This paper presents a novel dataset, SenseSeek, designed to evaluate the effectiveness of consumer-grade sensors in a complex information processing scenario: searching via systems (e.g., search engines), one of the common strategies users employ for information seeking. The SenseSeek dataset comprises data collected from 20 participants, 235 trials of the stimulated search process, 940 phases of stages in the search process, including the realization of Information Need (IN), Query Formulation (QF), Query Submission by Typing (QS-T) or Speaking (QS-S), and Relevance Judgment by Reading (RJ-R) or Listening (RJ-L). The data includes Electrodermal Activities (EDA), Electroencephalogram (EEG), PUPIL, GAZE, and MOTION data, which were captured using consumer-grade sensors. It also contains 258 features extracted from the sensor data, the gaze-annotated screen recordings, and task responses. We validate the usefulness of the dataset by providing baseline analysis on the impacts of different cognitive intents and interaction modalities on the sensor data, and effectiveness of the data in discriminating the search stages. To our knowledge, SenseSeek is the first dataset that characterizes the multiple stages involved in information seeking with physiological signals collected from multiple sensors. We hope this dataset can serve as a reference for future research on information-seeking behaviors.