From Optimizing Engagement to Measuring Value

TL;DR

This paper introduces a measurement-theoretic framework for recommendation systems that shifts focus from engagement prediction to directly measuring and optimizing for user value, with practical implementation on Twitter.

Contribution

It proposes a general latent variable model approach to operationalize and optimize for a user-defined notion of value in recommendation engines.

Findings

Successful implementation on Twitter with millions of users

Qualitative evaluation shows the model captures the intended value

Provides a normative and operational framework for value-based recommendations

Abstract

Most recommendation engines today are based on predicting user engagement, e.g. predicting whether a user will click on an item or not. However, there is potentially a large gap between engagement signals and a desired notion of "value" that is worth optimizing for. We use the framework of measurement theory to (a) confront the designer with a normative question about what the designer values, (b) provide a general latent variable model approach that can be used to operationalize the target construct and directly optimize for it, and (c) guide the designer in evaluating and revising their operationalization. We implement our approach on the Twitter platform on millions of users. In line with established approaches to assessing the validity of measurements, we perform a qualitative evaluation of how well our model captures a desired notion of "value".