Evaluation of Representation Models for Text Classification with AutoML Tools

TL;DR

This paper benchmarks different text representation methods and AutoML tools for text classification, finding that manual representations outperform auto-generated embeddings across multiple datasets.

Contribution

It provides a comparative analysis of manual and auto-generated text representations using open-source AutoML tools for text classification.

Findings

Manual text representations outperform AutoML-generated embeddings.

AutoML tools struggle with unstructured text data.

Benchmark includes four AutoML tools and eight datasets.

Abstract

Automated Machine Learning (AutoML) has gained increasing success on tabular data in recent years. However, processing unstructured data like text is a challenge and not widely supported by open-source AutoML tools. This work compares three manually created text representations and text embeddings automatically created by AutoML tools. Our benchmark includes four popular open-source AutoML tools and eight datasets for text classification purposes. The results show that straightforward text representations perform better than AutoML tools with automatically created text embeddings.