APS Explorer: Navigating Algorithm Performance Spaces for Informed Dataset Selection

TL;DR

The paper introduces APS Explorer, a web-based visualization tool that helps researchers explore Algorithm Performance Spaces interactively, improving dataset selection for recommender system experiments.

Contribution

It presents a novel interactive visualization tool for exploring Algorithm Performance Spaces to facilitate informed dataset selection.

Findings

Enables intuitive exploration of dataset similarities and differences.

Supports data-driven decisions in dataset selection.

Enhances reproducibility and reliability of recommender system experiments.

Abstract

Dataset selection is crucial for offline recommender system experiments, as mismatched data (e.g., sparse interaction scenarios require datasets with low user-item density) can lead to unreliable results. Yet, 86\% of ACM RecSys 2024 papers provide no justification for their dataset choices, with most relying on just four datasets: Amazon (38\%), MovieLens (34\%), Yelp (15\%), and Gowalla (12\%). While Algorithm Performance Spaces (APS) were proposed to guide dataset selection, their adoption has been limited due to the absence of an intuitive, interactive tool for APS exploration. Therefore, we introduce the APS Explorer, a web-based visualization tool for interactive APS exploration, enabling data-driven dataset selection. The APS Explorer provides three interactive features: (1) an interactive PCA plot showing dataset similarity via performance patterns, (2) a dynamic meta-feature table for dataset comparisons, and (3) a specialized visualization for pairwise algorithm performance.