An Analysis of Privacy-Aware Personalization Signals by Using Online Evaluation Methods

TL;DR

This paper investigates how to determine when personalization improves search results by analyzing user behavior and social network data through online evaluation methods, aiming to enhance real-time personalization.

Contribution

It introduces a novel online evaluation approach using team-draft interleaving to infer personalization signals from user interactions, surpassing survey-based methods.

Findings

Improved detection of personalization signals from user clicks

Enhanced real-time personalization capabilities

Better understanding of user preferences in search systems

Abstract

Personalization despite being an effective solution to the problem information overload remains tricky on account of multiple dimensions to consider. Furthermore, the challenge of avoiding overdoing personalization involves estimation of a user's preferences in relation to different queries. This work is an attempt to make inferences about when personalization would be beneficial by relating observable user behavior to his/her social network usage patterns and user-generated content. User behavior on a search system is observed by means of team-draft interleaving whereby results from two retrieval functions are presented in an interleaved manner, and user clicks are utilised to infer preference for a certain retrieval function. This improves upon earlier work which had limited usefulness due to reliance on user survey results; our findings may aid real-time personalization in search systems by detecting a user-related and query-related personalization signals.