Is deep learning necessary for simple classification tasks?

TL;DR

This paper compares AutoML and deep learning for binary classification, finding AutoML often outperforms simple DL, but combining both yields better results at the cost of increased training time.

Contribution

It provides a direct comparison of AutoML and DL on multiple datasets and introduces a new AutoML tool that integrates deep estimators.

Findings

AutoML outperforms simple DL classifiers on binary tasks.

Integrating DL into AutoML improves classification accuracy.

Training AutoML+DL pipelines requires substantial time.

Abstract

Automated machine learning (AutoML) and deep learning (DL) are two cutting-edge paradigms used to solve a myriad of inductive learning tasks. In spite of their successes, little guidance exists for when to choose one approach over the other in the context of specific real-world problems. Furthermore, relatively few tools exist that allow the integration of both AutoML and DL in the same analysis to yield results combining both of their strengths. Here, we seek to address both of these issues, by (1.) providing a head-to-head comparison of AutoML and DL in the context of binary classification on 6 well-characterized public datasets, and (2.) evaluating a new tool for genetic programming-based AutoML that incorporates deep estimators. Our observations suggest that AutoML outperforms simple DL classifiers when trained on similar datasets for binary classification but integrating DL into AutoML improves classification performance even further. However, the substantial time needed to train AutoML+DL pipelines will likely outweigh performance advantages in many applications.