Measuring the stability and plasticity of recommender systems

TL;DR

This paper introduces a new offline evaluation protocol to analyze the long-term stability and plasticity of recommender systems, addressing how models adapt or retain past patterns over time.

Contribution

It proposes a methodology to profile recommendation algorithms based on their ability to retain past data and adapt to new patterns, independent of datasets or metrics.

Findings

Different algorithms show distinct stability and plasticity profiles.

Preliminary results indicate a trade-off between stability and plasticity.

The framework is dataset-agnostic and applicable to various algorithms.

Abstract

The typical offline protocol to evaluate recommendation algorithms is to collect a dataset of user-item interactions and then use a part of this dataset to train a model, and the remaining data to measure how closely the model recommendations match the observed user interactions. This protocol is straightforward, useful and practical, but it only provides snapshot performance. We know, however, that online systems evolve over time. In general, it is a good idea that models are frequently retrained with recent data. But if this is the case, to what extent can we trust previous evaluations? How will a model perform when a different pattern (re)emerges? In this paper we propose a methodology to study how recommendation models behave when they are retrained. The idea is to profile algorithms according to their ability to, on the one hand, retain past patterns - stability - and, on the other hand, (quickly) adapt to changes - plasticity. We devise an offline evaluation protocol that provides detail on the long-term behavior of models, and that is agnostic to datasets, algorithms and metrics. To illustrate the potential of this framework, we present preliminary results of three different types of algorithms on the GoodReads dataset that suggest different stability and plasticity profiles depending on the algorithmic technique, and a possible trade-off between stability and plasticity. We further discuss the potential and limitations of the proposal and advance some possible improvements.