A Distributed and Accountable Approach to Offline Recommender Systems Evaluation

TL;DR

RecLab is an open source distributed framework that standardizes offline recommender system evaluation, enabling fair comparison, accountability, and easy integration of diverse algorithms through web protocols and a unified interface.

Contribution

It introduces RecLab, a novel open source tool that facilitates distributed, accountable, and standardized offline evaluation of recommender systems using web APIs.

Findings

Supports integration of algorithms with different technologies

Provides comprehensive metrics for evaluation

Stores results publicly for transparency and comparison

Abstract

Different software tools have been developed with the purpose of performing offline evaluations of recommender systems. However, the results obtained with these tools may be not directly comparable because of subtle differences in the experimental protocols and metrics. Furthermore, it is difficult to analyze in the same experimental conditions several algorithms without disclosing their implementation details. For these reasons, we introduce RecLab, an open source software for evaluating recommender systems in a distributed fashion. By relying on consolidated web protocols, we created RESTful APIs for training and querying recommenders remotely. In this way, it is possible to easily integrate into the same toolkit algorithms realized with different technologies. In details, the experimenter can perform an evaluation by simply visiting a web interface provided by RecLab. The framework will then interact with all the selected recommenders and it will compute and display a comprehensive set of measures, each representing a different metric. The results of all experiments are permanently stored and publicly available in order to support accountability and comparative analyses.