Deep Super Learner: A Deep Ensemble for Classification Problems

TL;DR

This paper introduces Deep Super Learner, a hierarchical ensemble method combining traditional machine learning algorithms, achieving competitive performance with deep neural networks while offering transparency, faster convergence, and fewer hyper-parameters.

Contribution

The paper presents a novel deep super learning approach that integrates traditional machine learning algorithms into a hierarchical ensemble, outperforming individual learners and some neural networks.

Findings

Deep super learner achieves competitive log loss and accuracy.

Outperforms individual base learners and single-layer ensembles.

Offers faster convergence and greater transparency on smaller datasets.

Abstract

Deep learning has become very popular for tasks such as predictive modeling and pattern recognition in handling big data. Deep learning is a powerful machine learning method that extracts lower level features and feeds them forward for the next layer to identify higher level features that improve performance. However, deep neural networks have drawbacks, which include many hyper-parameters and infinite architectures, opaqueness into results, and relatively slower convergence on smaller datasets. While traditional machine learning algorithms can address these drawbacks, they are not typically capable of the performance levels achieved by deep neural networks. To improve performance, ensemble methods are used to combine multiple base learners. Super learning is an ensemble that finds the optimal combination of diverse learning algorithms. This paper proposes deep super learning as an approach which achieves log loss and accuracy results competitive to deep neural networks while employing traditional machine learning algorithms in a hierarchical structure. The deep super learner is flexible, adaptable, and easy to train with good performance across different tasks using identical hyper-parameter values. Using traditional machine learning requires fewer hyper-parameters, allows transparency into results, and has relatively fast convergence on smaller datasets. Experimental results show that the deep super learner has superior performance compared to the individual base learners, single-layer ensembles, and in some cases deep neural networks. Performance of the deep super learner may further be improved with task-specific tuning.