Characterising Topic Familiarity and Query Specificity Using Eye-Tracking Data

TL;DR

This study demonstrates that eye-tracking data, specifically pupil dilation and gaze velocity, can effectively predict users' topic familiarity and query specificity without additional contextual information, using machine learning models.

Contribution

The paper introduces a novel approach using only pupil dilation and gaze velocity to infer topic familiarity and query specificity, along with a new annotation guideline for question answering queries.

Findings

Achieved 71.25% Macro F1 for topic familiarity prediction

Achieved 60.54% Macro F1 for query specificity prediction

Validated the feasibility of eye-tracking for understanding search behavior

Abstract

Eye-tracking data has been shown to correlate with a user's knowledge level and query formulation behaviour. While previous work has focused primarily on eye gaze fixations for attention analysis, often requiring additional contextual information, our study investigates the memory-related cognitive dimension by relying solely on pupil dilation and gaze velocity to infer users' topic familiarity and query specificity without needing any contextual information. Using eye-tracking data collected via a lab user study (N=18), we achieved a Macro F1 score of 71.25% for predicting topic familiarity with a Gradient Boosting classifier, and a Macro F1 score of 60.54% with a k-nearest neighbours (KNN) classifier for query specificity. Furthermore, we developed a novel annotation guideline -- specifically tailored for question answering -- to manually classify queries as Specific or Non-specific. This study demonstrates the feasibility of eye-tracking to better understand topic familiarity and query specificity in search.