Stochastic gradient descent with gradient estimator for categorical features

TL;DR

This paper introduces a novel gradient estimator tailored for sparse, one-hot encoded categorical data, improving optimization in machine learning models applied to such data types.

Contribution

It proposes a new gradient estimator specifically designed for sparse categorical features, enhancing model training and interpretability.

Findings

The new estimator outperforms existing ones on multiple datasets.

It improves model convergence and accuracy with categorical data.

A real-world retail dataset is released for further research.

Abstract

Categorical data are present in key areas such as health or supply chain, and this data require specific treatment. In order to apply recent machine learning models on such data, encoding is needed. In order to build interpretable models, one-hot encoding is still a very good solution, but such encoding creates sparse data. Gradient estimators are not suited for sparse data: the gradient is mainly considered as zero while it simply does not always exists, thus a novel gradient estimator is introduced. We show what this estimator minimizes in theory and show its efficiency on different datasets with multiple model architectures. This new estimator performs better than common estimators under similar settings. A real world retail dataset is also released after anonymization. Overall, the aim of this paper is to thoroughly consider categorical data and adapt models and optimizers to these key features.