Evaluating the Robustness of Learning from Implicit Feedback

TL;DR

This paper investigates how robust learning algorithms are when trained on implicit feedback in web search, using a user behavior model based on laboratory and real-world studies, and finds that such learning can be surprisingly resilient.

Contribution

It introduces a comprehensive user behavior model to evaluate the robustness of implicit feedback learning algorithms in web search.

Findings

Learning from implicit feedback is surprisingly robust across various user behaviors.

The algorithm remains effective in real-world search engine applications.

User behavior variability has limited impact on learning performance.

Abstract

This paper evaluates the robustness of learning from implicit feedback in web search. In particular, we create a model of user behavior by drawing upon user studies in laboratory and real-world settings. The model is used to understand the effect of user behavior on the performance of a learning algorithm for ranked retrieval. We explore a wide range of possible user behaviors and find that learning from implicit feedback can be surprisingly robust. This complements previous results that demonstrated our algorithm's effectiveness in a real-world search engine application.