Robustness of AutoML on Dirty Categorical Data

TL;DR

This paper evaluates how AutoML methods perform on dirty categorical data and proposes a pipeline with advanced encoding schemes to improve robustness and predictive accuracy.

Contribution

It introduces a pipeline transforming categorical data for AutoML and benchmarks its robustness on dirty datasets, providing insights into model performance and pipeline structures.

Findings

AutoML performance improves with advanced encoding schemes on dirty data

The proposed pipeline yields better predictive accuracy than standard AutoML approaches

Insights into AutoML pipeline structures on challenging datasets

Abstract

The goal of automated machine learning (AutoML) is to reduce trial and error when doing machine learning (ML). Although AutoML methods for classification are able to deal with data imperfections, such as outliers, multiple scales and missing data, their behavior is less known on dirty categorical datasets. These datasets often have several categorical features with high cardinality arising from issues such as lack of curation and automated collection. Recent research has shown that ML models can benefit from morphological encoders for dirty categorical data, leading to significantly superior predictive performance. However the effects of using such encoders in AutoML methods are not known at the moment. In this paper, we propose a pipeline that transforms categorical data into numerical data so that an AutoML can handle categorical data transformed by more advanced encoding schemes. We benchmark the current robustness of AutoML methods on a set of dirty datasets and compare it with the proposed pipeline. This allows us to get insight on differences in predictive performance. We also look at the ML pipelines built by AutoMLs in order to gain insight beyond the best model as typically returned by these methods.