Classifier Selection with Permutation Tests

TL;DR

This paper introduces a content-based recommender system for machine learning classifiers that uses permutation tests to better predict the most suitable classifier for a new dataset, improving recommendation quality.

Contribution

It proposes a novel approach utilizing permutation tests for classifier recommendation, enhancing the accuracy over traditional metrics like F-score.

Findings

Permutation tests improve classifier recommendation quality.

Extensive experiments with 8 classifiers and 65 datasets validate the approach.

Permutation-based assessment outperforms F-score in this context.

Abstract

This work presents a content-based recommender system for machine learning classifier algorithms. Given a new data set, a recommendation of what classifier is likely to perform best is made based on classifier performance over similar known data sets. This similarity is measured according to a data set characterization that includes several state-of-the-art metrics taking into account physical structure, statis- tics, and information theory. A novelty with respect to prior work is the use of a robust approach based on permutation tests to directly assess whether a given learning algorithm is able to exploit the attributes in a data set to predict class labels, and compare it to the more commonly used F-score metric for evalu- ating classifier performance. To evaluate our approach, we have conducted an extensive experimentation including 8 of the main machine learning classification methods with varying configurations and 65 bi- nary data sets, leading to over 2331 experiments. Our results show that using the information from the permutation test clearly improves the quality of the recommendations.