Comparison of Automated Machine Learning Tools for SMS Spam Message Filtering

TL;DR

This paper compares three AutoML tools for SMS spam filtering, finding ensemble models, especially H2O AutoML's Stacked Ensemble, outperform others in classification metrics, demonstrating AutoML's potential for effective spam detection.

Contribution

The study provides a performance comparison of three AutoML tools specifically for SMS spam filtering, highlighting the effectiveness of ensemble models and AutoML's utility in this domain.

Findings

H2O AutoML's Stacked Ensemble achieved the best performance.

Ensemble models outperformed individual models in classification.

AutoML tools can effectively automate SMS spam filtering.

Abstract

Short Message Service (SMS) is a very popular service used for communication by mobile users. However, this popular service can be abused by executing illegal activities and influencing security risks. Nowadays, many automatic machine learning (AutoML) tools exist which can help domain experts and lay users to build high-quality ML models with little or no machine learning knowledge. In this work, a classification performance comparison was conducted between three automatic ML tools for SMS spam message filtering. These tools are mljar-supervised AutoML, H2O AutoML, and Tree-based Pipeline Optimization Tool (TPOT) AutoML. Experimental results showed that ensemble models achieved the best classification performance. The Stacked Ensemble model, which was built using H2O AutoML, achieved the best performance in terms of Log Loss (0.8370), true positive (1088/1116), and true negative (281/287) metrics. There is a 19.05\% improvement in Log Loss with respect to TPOT AutoML and 5.56\% improvement with respect to mljar-supervised AutoML. The satisfactory filtering performance achieved with AutoML tools provides a potential application for AutoML tools to automatically determine the best ML model that can perform best for SMS spam message filtering.