Predicting User Knowledge Gain in Informational Search Sessions

TL;DR

This paper presents a supervised model that predicts user knowledge gain during informational search sessions, using features from real-world search behavior and calibrated knowledge tests, to enhance understanding of learning outcomes in web search.

Contribution

Introduces a novel supervised approach to predict user knowledge gain from search session features, validated with real-world data and scientifically calibrated knowledge assessments.

Findings

Supervised models accurately predict knowledge gain from search session features.

Features derived from search behavior outperform baseline models.

Model demonstrates potential to improve search engines for learning outcomes.

Abstract

Web search is frequently used by people to acquire new knowledge and to satisfy learning-related objectives. In this context, informational search missions with an intention to obtain knowledge pertaining to a topic are prominent. The importance of learning as an outcome of web search has been recognized. Yet, there is a lack of understanding of the impact of web search on a user's knowledge state. Predicting the knowledge gain of users can be an important step forward if web search engines that are currently optimized for relevance can be molded to serve learning outcomes. In this paper, we introduce a supervised model to predict a user's knowledge state and knowledge gain from features captured during the search sessions. To measure and predict the knowledge gain of users in informational search sessions, we recruited 468 distinct users using crowdsourcing and orchestrated real-world search sessions spanning 11 different topics and information needs. By using scientifically formulated knowledge tests, we calibrated the knowledge of users before and after their search sessions, quantifying their knowledge gain. Our supervised models utilise and derive a comprehensive set of features from the current state of the art and compare performance of a range of feature sets and feature selection strategies. Through our results, we demonstrate the ability to predict and classify the knowledge state and gain using features obtained during search sessions, exhibiting superior performance to an existing baseline in the knowledge state prediction task.