Beyond NDCG: behavioral testing of recommender systems with RecList

TL;DR

RecList is a new behavioral testing framework for recommender systems that complements traditional metrics, enabling more nuanced evaluation of real-world performance across different use cases.

Contribution

It introduces a scalable, plug-and-play methodology for behavioral testing of recommender systems, applicable to both algorithms and commercial black-box systems.

Findings

RecList effectively analyzes known algorithms and commercial systems.

It reveals nuanced behaviors not captured by standard metrics.

Open source release facilitates community adoption.

Abstract

As with most Machine Learning systems, recommender systems are typically evaluated through performance metrics computed over held-out data points. However, real-world behavior is undoubtedly nuanced: ad hoc error analysis and deployment-specific tests must be employed to ensure the desired quality in actual deployments. In this paper, we propose RecList, a behavioral-based testing methodology. RecList organizes recommender systems by use case and introduces a general plug-and-play procedure to scale up behavioral testing. We demonstrate its capabilities by analyzing known algorithms and black-box commercial systems, and we release RecList as an open source, extensible package for the community.