Quantile Encoder: Tackling High Cardinality Categorical Features in Regression Problems

TL;DR

This paper introduces the Quantile Encoder, a novel method for encoding high-cardinality categorical features in regression tasks, which improves performance especially with skewed distributions by using quantiles and smoothing.

Contribution

The paper proposes a specialized quantile-based encoding technique for categorical features in regression, outperforming traditional encoders and including methods to reduce overfitting.

Findings

Outperforms mean target encoder in MAE

Effective with skewed and long-tailed distributions

Enhanced with feature expansion using multiple quantiles

Abstract

Regression problems have been widely studied in machinelearning literature resulting in a plethora of regression models and performance measures. However, there are few techniques specially dedicated to solve the problem of how to incorporate categorical features to regression problems. Usually, categorical feature encoders are general enough to cover both classification and regression problems. This lack of specificity results in underperforming regression models. In this paper,we provide an in-depth analysis of how to tackle high cardinality categor-ical features with the quantile. Our proposal outperforms state-of-the-encoders, including the traditional statistical mean target encoder, when considering the Mean Absolute Error, especially in the presence of long-tailed or skewed distributions. Besides, to deal with possible overfitting when there are categories with small support, our encoder benefits from additive smoothing. Finally, we describe how to expand the encoded values by creating a set of features with different quantiles. This expanded encoder provides a more informative output about the categorical feature in question, further boosting the performance of the regression model.