Interactive Modeling of Concept Drift and Errors in Relevance Feedback

TL;DR

This paper introduces a Bayesian model and timeline interface for interactive relevance feedback, effectively handling noisy user input and concept drift in exploratory search, improving user correction and discovery.

Contribution

It presents a novel Bayesian feedback accuracy model combined with a timeline visualization, enabling better detection and correction of feedback errors during search.

Findings

Model outperforms simpler baselines in simulations

Interface helps users identify and correct feedback mistakes

Approaches oracle performance with minimal user interaction

Abstract

Users giving relevance feedback in exploratory search are often uncertain about the correctness of their feedback, which may result in noisy or even erroneous feedback. Additionally, the search intent of the user may be volatile as the user is constantly learning and reformulating her search hypotheses during the search. This may lead to a noticeable concept drift in the feedback. We formulate a Bayesian regression model for predicting the accuracy of each individual user feedback and thus find outliers in the feedback data set. Additionally, we introduce a timeline interface that visualizes the feedback history to the user and gives her suggestions on which past feedback is likely in need of adjustment. This interface also allows the user to adjust the feedback accuracy inferences made by the model. Simulation experiments demonstrate that the performance of the new user model outperforms a simpler baseline and that the performance approaches that of an oracle, given a small amount of additional user interaction. A user study shows that the proposed modelling technique, combined with the timeline interface, makes it easier for the users to notice and correct mistakes in their feedback, and to discover new items.