Comparison of 14 different families of classification algorithms on 115 binary datasets

TL;DR

This study systematically compares 14 classification algorithms across 115 binary datasets, finding that random forest, gradient boosting, and RBF SVM perform similarly with negligible practical differences, and RBF SVM is the fastest.

Contribution

It provides a comprehensive empirical comparison of diverse classifiers on real datasets, highlighting practical equivalences and efficiency considerations.

Findings

Random forest, gbm, and RBF SVM are statistically similar in performance.

A change of less than 0.0112 in error rate is practically irrelevant.

RBF SVM is the fastest classifier in training and testing.

Abstract

We tested 14 very different classification algorithms (random forest, gradient boosting machines, SVM - linear, polynomial, and RBF - 1-hidden-layer neural nets, extreme learning machines, k-nearest neighbors and a bagging of knn, naive Bayes, learning vector quantization, elastic net logistic regression, sparse linear discriminant analysis, and a boosting of linear classifiers) on 115 real life binary datasets. We followed the Demsar analysis and found that the three best classifiers (random forest, gbm and RBF SVM) are not significantly different from each other. We also discuss that a change of less then 0.0112 in the error rate should be considered as an irrelevant change, and used a Bayesian ANOVA analysis to conclude that with high probability the differences between these three classifiers is not of practical consequence. We also verified the execution time of "standard implementations" of these algorithms and concluded that RBF SVM is the fastest (significantly so) both in training time and in training plus testing time.