Limits to Surprise in Recommender Systems

TL;DR

This paper introduces a novel 'normalized surprise' metric for recommender systems, quantifying the maximum and minimum potential surprise to evaluate how well algorithms embed surprise within their recommendations.

Contribution

The study proposes a new theoretical framework and metric for measuring the limits of surprise in recommender systems, validated through experiments on MovieLens data.

Findings

The normalized surprise metric effectively measures surprise embedding capacity.

The metric is consistent across different algorithms and data representations.

Experimental results support the theoretical limits of surprise in recommendations.

Abstract

In this study, we address the challenge of measuring the ability of a recommender system to make surprising recommendations. Although current evaluation methods make it possible to determine if two algorithms can make recommendations with a significant difference in their average surprise measure, it could be of interest to our community to know how competent an algorithm is at embedding surprise in its recommendations, without having to resort to making a direct comparison with another algorithm. We argue that a) surprise is a finite resource in a recommender system, b) there is a limit to how much surprise any algorithm can embed in a recommendation, and c) this limit can provide us with a scale against which the performance of any algorithm can be measured. By exploring these ideas, it is possible to define the concepts of maximum and minimum potential surprise and design a surprise metric called "normalised surprise" that employs these limits to potential surprise. Two experiments were conducted to test the proposed metric. The aim of the first was to validate the quality of the estimates of minimum and maximum potential surprise produced by a greedy algorithm. The purpose of the second experiment was to analyse the behaviour of the proposed metric using the MovieLens dataset. The results confirmed the behaviour that was expected, and showed that the proposed surprise metric is both effective and consistent for differing choices of recommendation algorithms, data representations and distance functions.