Sometimes You Want to Go Where Everybody Knows your Name

TL;DR

This paper proposes a new metric for model personalization that balances user-specific performance with global data, aiming to improve personalization without compromising privacy, demonstrated through a sentiment classification experiment.

Contribution

It introduces a novel metric for personalization that incorporates privacy constraints and provides an experimental framework to evaluate the trade-off between global and individual performance.

Findings

The new metric effectively measures personalization quality.

The experiment illustrates the tension between global and user-specific performance.

The approach respects user privacy by avoiding data centralization.

Abstract

We introduce a new metric for measuring how well a model personalizes to a user's specific preferences. We define personalization as a weighting between performance on user specific data and performance on a more general global dataset that represents many different users. This global term serves as a form of regularization that forces us to not overfit to individual users who have small amounts of data. In order to protect user privacy, we add the constraint that we may not centralize or share user data. We also contribute a simple experiment in which we simulate classifying sentiment for users with very distinct vocabularies. This experiment functions as an example of the tension between doing well globally on all users, and doing well on any specific individual user. It also provides a concrete example of how to employ our new metric to help reason about and resolve this tension. We hope this work can help frame and ground future work into personalization.