VAR: Visual Analysis for Rashomon Set of Machine Learning Models' Performance

TL;DR

The paper introduces VAR, a visualization tool combining heatmaps and scatter plots, to compare machine learning models within the Rashomon set, aiding in model selection and understanding model diversity.

Contribution

It presents a novel visualization method for horizontal comparison of models in the Rashomon set, addressing a gap in existing analysis techniques.

Findings

Enables effective comparison of models within the Rashomon set

Helps identify optimal models under specific conditions

Provides insights into the diversity of models with similar accuracy

Abstract

Evaluating the performance of closely matched machine learning(ML) models under specific conditions has long been a focus of researchers in the field of machine learning. The Rashomon set is a collection of closely matched ML models, encompassing a wide range of models with similar accuracies but different structures. Traditionally, the analysis of these sets has focused on vertical structural analysis, which involves comparing the corresponding features at various levels within the ML models. However, there has been a lack of effective visualization methods for horizontally comparing multiple models with specific features. We propose the VAR visualization solution. VAR uses visualization to perform comparisons of ML models within the Rashomon set. This solution combines heatmaps and scatter plots to facilitate the comparison. With the help of VAR, ML model developers can identify the optimal model under specific conditions and better understand the Rashomon set's overall characteristics.