Evaluating the performance-deviation of itemKNN in RecBole and LensKit

TL;DR

This paper compares the performance of itemKNN algorithms in RecBole and LensKit libraries across multiple datasets, identifying key differences in similarity calculations that affect accuracy metrics and demonstrating how adjustments can align their results.

Contribution

It provides a detailed analysis of performance deviations between RecBole and LensKit's itemKNN implementations and proposes modifications to standardize their comparison.

Findings

RecBole outperforms LensKit on ML-100K in nDCG, precision, and recall.

Aligning nDCG calculations reduces performance differences.

Similarity matrix calculation differences are the main cause of deviations.

Abstract

This study examines the performance of item-based k-Nearest Neighbors (ItemKNN) algorithms in the RecBole and LensKit recommender system libraries. Using four data sets (Anime, Modcloth, ML-100K, and ML-1M), we assess each library's efficiency, accuracy, and scalability, focusing primarily on normalized discounted cumulative gain (nDCG). Our results show that RecBole outperforms LensKit on two of three metrics on the ML-100K data set: it achieved an 18% higher nDCG, 14% higher precision, and 35% lower recall. To ensure a fair comparison, we adjusted LensKit's nDCG calculation to match RecBole's method. This alignment made the performance more comparable, with LensKit achieving an nDCG of 0.2540 and RecBole 0.2674. Differences in similarity matrix calculations were identified as the main cause of performance deviations. After modifying LensKit to retain only the top K similar items, both libraries showed nearly identical nDCG values across all data sets. For instance, both achieved an nDCG of 0.2586 on the ML-1M data set with the same random seed. Initially, LensKit's original implementation only surpassed RecBole in the ModCloth dataset.