Measuring the User Satisfaction in a Recommendation Interface with Multiple Carousels

TL;DR

This paper introduces a new two-dimensional evaluation protocol for recommendation carousels that considers the interaction between multiple carousels and user position bias, improving the assessment of user satisfaction.

Contribution

It proposes a novel evaluation method that accounts for the presence of multiple carousels and their relative positions, addressing limitations of traditional isolated evaluation protocols.

Findings

Traditional evaluation protocols ignore carousel interactions and position bias.

The new protocol better predicts user satisfaction in multi-carousel interfaces.

Experiment results show different criteria are preferred for recommendations in carousel settings.

Abstract

It is common for video-on-demand and music streaming services to adopt a user interface composed of several recommendation lists, i.e. widgets or swipeable carousels, each generated according to a specific criterion or algorithm (e.g. most recent, top popular, recommended for you, editors' choice, etc.). Selecting the appropriate combination of carousel has significant impact on user satisfaction. A crucial aspect of this user interface is that to measure the relevance a new carousel for the user it is not sufficient to account solely for its individual quality. Instead, it should be considered that other carousels will already be present in the interface. This is not considered by traditional evaluation protocols for recommenders systems, in which each carousel is evaluated in isolation, regardless of (i) which other carousels are displayed to the user and (ii) the relative position of the carousel with respect to other carousels. Hence, we propose a two-dimensional evaluation protocol for a carousel setting that will measure the quality of a recommendation carousel based on how much it improves upon the quality of an already available set of carousels. Our evaluation protocol takes into account also the position bias, i.e. users do not explore the carousels sequentially, but rather concentrate on the top-left corner of the screen. We report experiments on the movie domain and notice that under a carousel setting the definition of which criteria has to be preferred to generate a list of recommended items changes with respect to what is commonly understood.