Efficient Representations for High-Cardinality Categorical Variables in Machine Learning

TL;DR

This paper proposes new encoding methods for high-cardinality categorical variables that improve model performance and computational efficiency by generating compact, informative embeddings, addressing limitations of traditional one-hot encoding.

Contribution

Introduction of mean, low-rank, and multinomial logistic regression encoding techniques that provide scalable and effective representations for large categorical datasets.

Findings

Significant performance improvements over baseline methods

Enhanced computational efficiency in large datasets

Effective in diverse application domains

Abstract

High\-cardinality categorical variables pose significant challenges in machine learning, particularly in terms of computational efficiency and model interpretability. Traditional one\-hot encoding often results in high\-dimensional sparse feature spaces, increasing the risk of overfitting and reducing scalability. This paper introduces novel encoding techniques, including means encoding, low\-rank encoding, and multinomial logistic regression encoding, to address these challenges. These methods leverage sufficient representations to generate compact and informative embeddings of categorical data. We conduct rigorous theoretical analyses and empirical validations on diverse datasets, demonstrating significant improvements in model performance and computational efficiency compared to baseline methods. The proposed techniques are particularly effective in domains requiring scalable solutions for large datasets, paving the way for more robust and efficient applications in machine learning.