Encoding high-cardinality string categorical variables

TL;DR

This paper introduces scalable, interpretable encoding methods for high-cardinality string categorical variables, improving supervised learning without extensive feature engineering or data cleaning.

Contribution

It proposes two novel encoding approaches—Gamma-Poisson matrix factorization and min-hash encoding—for efficient, interpretable, and scalable handling of string categories in statistical models.

Findings

Both methods are scalable and streamable.

They improve supervised learning performance.

Min-hash is best for scalability; Gamma-Poisson for interpretability.

Abstract

Statistical models usually require vector representations of categorical variables, using for instance one-hot encoding. This strategy breaks down when the number of categories grows, as it creates high-dimensional feature vectors. Additionally, for string entries, one-hot encoding does not capture information in their representation.Here, we seek low-dimensional encoding of high-cardinality string categorical variables. Ideally, these should be: scalable to many categories; interpretable to end users; and facilitate statistical analysis. We introduce two encoding approaches for string categories: a Gamma-Poisson matrix factorization on substring counts, and the min-hash encoder, for fast approximation of string similarities. We show that min-hash turns set inclusions into inequality relations that are easier to learn. Both approaches are scalable and streamable. Experiments on real and simulated data show that these methods improve supervised learning with high-cardinality categorical variables. We recommend the following: if scalability is central, the min-hash encoder is the best option as it does not require any data fit; if interpretability is important, the Gamma-Poisson factorization is the best alternative, as it can be interpreted as one-hot encoding on inferred categories with informative feature names. Both models enable autoML on the original string entries as they remove the need for feature engineering or data cleaning.

Tables

Table 1. TABLE I : Examples of high-cardinality categorical variables.

Count	Non Proprietary Name
1736	alcohol
1089	ethyl alcohol
556	isopropyl alcohol
16	polyvinyl alcohol
12	isopropyl alcohol swab
12	62% ethyl alcohol
6	alcohol 68%
6	alcohol denat
5	dehydrated alcohol

Table 2. (a) Count for some of the categories containing the word alcohol in the Drug Directory dataset. The dataset contains more than 120k samples.

Count	Non Proprietary Name
1736	alcohol
1089	ethyl alcohol
556	isopropyl alcohol
16	polyvinyl alcohol
12	isopropyl alcohol swab
12	62% ethyl alcohol
6	alcohol 68%
6	alcohol denat
5	dehydrated alcohol

Table 3. (b) Some categories in the Employee Salaries dataset. For 10 000 employees, there are almost 400 different occupations. Yet, they share relevant substrings.

Employee Position Title
Police Aide
Master Police Officer
Mechanic Technician II
Police Officer III
Senior Architect
Senior Engineer Technician
Social Worker III
Bus Operator

Table 4. TABLE II : Summary of notations

Symbol	Definition
$𝒮$	Set of all finite-length strings.
$𝒢\left(s\right)\subseteq 𝒮$	Set of all consecutive n-grams in $s\in 𝒮$.
$𝒱={\bigcup }_{i=1}^n𝒢\left(s_i\right)$	Vocabulary of n-grams in the train set.
$C$	Categorical variable.
$n$	Number of samples.
$d$	Dimension of the categorical encoder.
$m=\left|𝒱\right|$	Cardinality of the vocabulary.
$𝐅\in {ℝ}^{n\times m}$	Count matrix of n-grams.
$𝐗\in {ℝ}^{n\times d}$	Feature matrix of $C$.
$\text{sim}:𝒮\times 𝒮\to [0,1]$	String similarity.
$h_k:𝒮\to [0,1]$	Hash function with salt value equal to $k$.
$Z_k:𝒮\to [0,1]$	Min-hash function with salt value equal to $k$.

Table 5. TABLE III : Non-curated datasets. Description for the corresponding high-cardinality categorical variable.

Dataset	#samples	#categories	#categories per 1000 samples	Gini coefficient	Mean category length (#chars)	Source of high cardinality
Crime Data	1.5M	135	64.5	0.85	30.6	Multi-label
Medical Charges	163k	100	99.9	0.23	41.1	Multi-label
Kickstarter Projects	281k	158	123.8	0.64	11.0	Multi-label
Employee Salaries	9.2k	385	186.3	0.79	24.9	Multi-label
Open Payments	2.0M	1.4k	231.9	0.90	24.7	Multi-label
Traffic Violations	1.2M	11.3k	243.5	0.97	62.1	Typos; Description
Vancouver Employees	2.6k	640	341.8	0.67	21.5	Multi-label
Federal Election	3.3M	145.3k	361.7	0.76	13.0	Typos; Multi-label
Midwest Survey	2.8k	844	371.9	0.67	15.0	Typos
Met Objects	469k	26.8k	386.1	0.88	12.2	Typos; Multi-label
Drug Directory	120k	17.1k	641.9	0.81	31.3	Multi-label
Road Safety	139k	15.8k	790.1	0.65	29.0	Multi-label
Public Procurement	352k	28.9k	804.6	0.82	46.8	Multi-label; Multi-language
Journal Influence	3.6k	3.2k	956.9	0.10	30.0	Multi-label; Multi-language
Building Permits	554k	430.6k	940.0	0.48	94.0	Typos; Description
Wine Reviews	138k	89.1k	997.7	0.23	245.0	Description
Colleges	7.8k	6.9k	998.0	0.02	32.1	Multi-label

Table 6. TABLE IV : Examples of simulated categorical variables.

Type	Example categories
Ground truth	chicken; eagle; giraffe; horse; leopard;
	lion; tiger; turtle.
Multi-label	lion chicken; horse eagle lion.
Typos (10%)	itger; tiuger; tgier; tiegr; tigre; ttiger.

Table 7. TABLE V : Recovery of categories for simulations : Normalized mutual information (NMI) for different encoders.

Encoder	Multi-label			Typos
	$d$=6	$𝐝$=8	$d$=10	$d$=6	$𝐝$=8	$d$=10
Tf-idf + SVD	0.16	0.18	0.17	0.17	0.16	0.16
FastText + SVD	0.08	0.09	0.09	0.07	0.08	0.08
Bert + SVD	0.03	0.03	0.03	0.05	0.06	0.06
Similarity Encoder	0.32	0.25	0.24	0.72	0.82	0.80
Min-hash Encoder	0.14	0.15	0.13	0.14	0.15	0.13
Gamma-Poisson	0.76	0.82	0.79	0.77	0.83	0.80

Table 8. TABLE VI : Recovering true categories for curated entries. NMI for different encoders ( d 𝑑 d =30) – Appendix C gives results for different dimensions.

Dataset	Gamma	Similarity	Tf-idf	FastText	Bert
(cardinality)	Poisson	Encoding	+ SVD	+ SVD	+ SVD
Adult (15)	0.75	0.71	0.54	0.19	0.07
Cacao Flavors (100)	0.51	0.30	0.28	0.07	0.04
California Housing (5)	0.46	0.51	0.56	0.20	0.05
Dating Profiles (19)	0.52	0.24	0.25	0.12	0.05
House Prices (15)	0.83	0.25	0.32	0.11	0.05
House Sales (70)	0.42	0.04	0.18	0.06	0.02
Intrusion Detection (66)	0.34	0.58	0.46	0.11	0.05

Table 9. TABLE VII : Comparing SVD and Gaussian random projection as a dimensionality reduction Wilcoxon test p-values for different encoders. Prediction performance with SVD is significantly superior for tf-idf, FastText and Bert.

Encoder	SVD v/s Random projection (p-value)
Tf-idf	0.001
FastText	0.006
Bert	0.001
One-hot	0.717

Table 10. TABLE VIII : Parameter values for the Gamma-Poisson factorization. The same parameters were used for all datasets.

Parameter	Definition	Default value
${\alpha }_i$	Poisson shape	1.1
${\beta }_i$	Poisson scale	1.0
$\rho$	Discount factor	0.95
$q$	Mini-batch size	256
$\eta$	Approximation error	${10}^{-4}$
$ϵ$	Approximation error	${10}^{-3}$

Table 11. TABLE XI : Recovering true categories for curated categorical variables. NMI for different encoders ( d = | C | 𝑑 𝐶 d{=}|C| ).

Dataset	Gamma-	Similarity	Tf-idf	FastText	Bert
(cardinality)	Poisson	Encoding	+ SVD	+ SVD	+ SVD
Adult (15)	0.84	0.71	0.54	0.19	0.07
Cacao Flavors (100)	0.48	0.34	0.34	0.1	0.05
California Housing (5)	0.83	0.51	0.56	0.20	0.05
Dating Profiles (19)	0.47	0.26	0.29	0.12	0.06
House Prices (15)	0.91	0.25	0.32	0.11	0.05
House Sales (70)	0.29	0.03	0.26	0.07	0.03
Intrusion Detection (66)	0.27	0.65	0.61	0.13	0.06

Table 12. TABLE XII : Recovering true categories for curated entries. NMI for different encoders ( d 𝑑 d =10).

Dataset	Gamma	Similarity	Tf-idf	FastText	Bert
(cardinality)	Poisson	Encoding	+ SVD	+ SVD	+ SVD
Adult (15)	0.73	0.61	0.41	0.14	0.05
Cacao Flavors (100)	0.44	0.28	0.21	0.05	0.03
California Housing (5)	0.63	0.51	0.56	0.20	0.05
Dating Profiles (19)	0.34	0.28	0.20	0.08	0.03
House Prices (15)	0.81	0.26	0.26	0.09	0.04
House Sales (70)	0.49	0.04	0.11	0.05	0.02
Intrusion Detection (66)	0.34	0.53	0.46	0.08	0.04

Table 13. TABLE XIII : Recovering true categories for curated entries. NMI for different encoders ( d 𝑑 d =100).

Dataset	Gamma	Similarity	Tf-idf	FastText	Bert
(cardinality)	Poisson	Encoding	+ SVD	+ SVD	+ SVD
Adult (15)	0.55	0.71	0.54	0.19	0.06
Cacao Flavors (100)	0.47	0.34	0.34	0.10	0.05
California Housing (5)	0.18	0.51	0.56	0.20	0.05
Dating Profiles (19)	0.30	0.26	0.29	0.12	0.06
House Prices (15)	0.63	0.25	0.32	0.11	0.05
House Sales (70)	0.21	0.03	0.26	0.07	0.03
Intrusion Detection (66)	0.23	0.65	0.61	0.13	0.06

Equations

$${\mathbf{x}}_i^{\text{sim}}\stackrel{\text{def}}{=}[\text{sim}(s_i,s^{(1)}),\text{sim}(s_i,s^{(2)}),\dots ,\text{sim}(s_i,s^{(k)})]\in {\mathbb{R}}^k,$$

$${\text{sim}}_{\text{one-hot}}(s_i,s_j)=\text{mathbbm}1[s_i=s_j],$$

$${\text{sim}}_{\text{n-gram}}(s_i,s_j)=J(\mathcal{G}(s_i),\mathcal{G}(s_j))=\frac{|\mathcal{G}(s_i)\cap \mathcal{G}(s_j)|}{|\mathcal{G}(s_i)\cup \mathcal{G}(s_j)|}$$

$$Z(\mathcal{X})\stackrel{\text{def}}{=}\underset{x\in \mathcal{X}}{\min }\pi (x)$$

$$\mathbb{P}\left(Z(\mathcal{X})=Z(\mathcal{Y})\right)=\frac{|\mathcal{X}\cap \mathcal{Y}|}{|\mathcal{X}\cup \mathcal{Y}|}=J(\mathcal{X},\mathcal{Y})$$

$$\sum _{j=1}^d\text{mathbbm}1[Z_j(\mathcal{X})=Z_j(\mathcal{Y})]\sim \mathcal{B}(d,J(\mathcal{X},\mathcal{Y})).$$

$$\mathbb{P}\left(Z(\mathcal{Y})-Z(\mathcal{X})\le 0|\mathcal{X}\cap \mathcal{Y}=\varnothing \right)$$

$$\mathcal{X}⊈\mathcal{Y}\Rightarrow \mathbb{P}\left(\sum _{j=1}^d\text{mathbbm}1[Z_j(\mathcal{Y})\le Z_j(\mathcal{X})]=d\right)\le ϵ.$$

$$\mathcal{X}⊈\mathcal{Y}⟺\exists k\in \mathbb{N},0\le k<k_x\text{ such that }|\mathcal{X}\cap \mathcal{Y}|=k$$

$$\mathbb{P}\left(Z(\mathcal{Y})\le Z(\mathcal{X})|\mathcal{X}⊈\mathcal{Y}\right)$$

$$=(k_y-k)/(k_x+k_y-k)$$

$$\le k_y/(k_x+k_y)$$

$$=\mathbb{P}\left(Z(\mathcal{Y})\le Z(\mathcal{X})|\mathcal{X}\cap \mathcal{Y}=\varnothing \right)$$

$$\mathbb{P}\left({\sum }_{j=1}^d\text{mathbbm}1[Z_j(\mathcal{Y})\le Z_j(\mathcal{X})]=d|\mathcal{X}⊈\mathcal{Y}\right)$$

$$Z_j(\mathcal{X})=\underset{x\in \mathcal{X}}{\min }{h}_j(x),$$

$${\mathbf{x}}^{\text{min-hash}}(s)\stackrel{\text{def}}{=}[Z_1(\mathcal{G}(s)),\dots ,Z_d(\mathcal{G}(s))]\in {\mathbb{R}}^d.$$

$$\frac{1}{d}\mathbb{E}\left[\Vert {\mathbf{x}}^{\text{min-hash}}(s_i)-{\mathbf{x}}^{\text{min-hash}}(s_j){\Vert }_{{\ell }_0}\right]=J(\mathcal{G}(s_i),\mathcal{G}(s_j))$$

$$\mathbf{f}\approx \mathbf{x}\mathbf{\Lambda },$$

$$\mathbf{F}\approx \mathbf{X}\mathbf{\Lambda },\text{with }\mathbf{X}\in {\mathbb{R}}^{n\times d},\mathbf{\Lambda }\in {\mathbb{R}}^{d\times m}$$

$$p\bigl{(}f_{j}|(\mathbf{x}\,\mathbf{\Lambda})_{j}\bigr{)}=\frac{1}{f_{j}!}(\mathbf{x}\,\mathbf{\Lambda})_{j}^{f_{j}}e^{-(\mathbf{x}\,\mathbf{\Lambda})_{j}},\quad j=1,...,m.$$

$$p(x_i)=\frac{x_i^{{\alpha }_i-1}{e}^{-x_i/{\beta }_i}}{{\beta }_i^{{\alpha }_i}\Gamma ({\alpha }_i)},i=1,...,d,$$

$$\log \mathcal{L}$$

$$\sum _{i=1}^d({\alpha }_i-1)\log (x_i)-\frac{x_i}{{\beta }_i}-{\alpha }_i\log {\beta }_i-\log \Gamma ({\alpha }_i)$$

$$\frac{\partial }{\partial {\Lambda }_{ij}}\log \mathcal{L}$$

$$\frac{\partial }{\partial x_i}\log \mathcal{L}$$

$$\left(\begin{array}{c}\mathbf{F}\\ \text{diag}(\mathbf{\beta }{)}^{-1}\end{array}\right)=\mathbf{X}\left(\begin{array}{c}\mathbf{\Lambda }\\ \text{diag}(\mathbf{\alpha })-I_d\end{array}\right)$$

$${\Lambda }_{ij}$$

$$x_{\ell i}$$

$$\text{NMI}=2\frac{I(X_1;X_2)}{H(X_1)+H(X_2)}$$

$${\text{relative score}}_j^i=100\frac{{\text{score}}_j^i-{\min }_j{\text{score}}_j^i}{{\max }_j{\text{score}}_j^i-{\min }_j{\text{score}}_j^i}$$

Full text

Here, we seek low-dimensional encoding of high-cardinality string categorical variables. Ideally, these should be: scalable to many categories; interpretable to end users; and facilitate statistical analysis. We introduce two encoding approaches for string categories: a Gamma-Poisson matrix factorization on substring counts, and a min-hash encoder, for fast approximation of string similarities. We show that min-hash turns set inclusions into inequality relations that are easier to learn. Both approaches are scalable and streamable. Experiments on real and simulated data show that these methods improve supervised learning with high-cardinality categorical variables. We recommend the following: if scalability is central, the min-hash encoder is the best option as it does not require any data fit; if interpretability is important, the Gamma-Poisson factorization is the best alternative, as it can be interpreted as one-hot encoding on inferred categories with informative feature names. Both models enable autoML on string entries as they remove the need for feature engineering or data cleaning.

1 Introduction

Tabular datasets often contain columns with string entries. However, fitting statistical models on such data generally requires a numerical representation of all entries, which calls for building an encoding, or vector representation of the entries. Considering string entries as nominal—unordered—categories gives well-framed statistical analysis. In such situations, categories are assumed to be mutually exclusive and unrelated, with a fixed known set of possible values. Yet, in many real-world datasets, string columns are not standardized in a small number of categories. This poses challenges for statistical analysis. First, the set of all possible categories may be huge and not known a priori, as the number of different strings in the column can indefinitely increase with the number of samples. Second, categories may be related: they often carry some morphological or semantic links.

The classic approach to encode categorical variables for statistical analysis is one-hot encoding. It creates vectors that agree with the general intuition of nominal categories: orthogonal and equidistant [1]. However, for high-cardinality categories, one-hot encoding leads to feature vectors of high dimensionality. This is especially problematic in big data settings, which can lead to a very large number of categories, posing computational and statistical problems.

Data engineering practices typically tackle these issues with data-cleaning techniques [2, 3]. In particular, deduplication tries to merge different variants of the same entity [4, 5, 6]. A related concept is that of normalization, used in databases and text processing to put entries in canonical forms. However, data cleaning or normalization often requires human intervention, and are major costs in data analysis111 Kaggle industry survey: https://www.kaggle.com/surveys/2017. To avoid the cleaning step, Similarity encoding [7] relaxes one-hot encoding by using string similarities [8]. Hence, it addresses the problem of related categories and has been shown to improve statistical analysis upon one-hot encoding [7]. Yet, it does not tackle the problem of high cardinality, and the data analyst much resort to heuristics such as choosing a subset of the training categories [7].

Here, we seek encoding approaches for statistical analysis on string categorical entries that are suited to a very large number of categories without any human intervention: avoiding data cleaning, feature engineering, or neural architecture search. Our goals are: i) to provide feature vectors of limited dimensionality without any cleaning or feature engineering step, even for very large datasets; ii) to improve statistical analysis tasks such as supervised learning; and iii) to preserve the intuitions behind categories: entries can be arranged in natural groups that can be easily interpreted. We study two novel encoding methods that both address scalability and statistical performance: a min-hash encoder, based on locality-sensitive hashing (LSH) [9], and a low-rank model of co-occurrences in character n-grams: a Gamma-Poisson matrix factorization, suited to counting statistics. Both models scale linearly with the number of samples and are suitable for statistical analysis in streaming settings. Moreover, we show that the Gamma-Poisson factorization model enables interpretability with a sparse encoding that expresses the entries of the data as linear combinations of a small number of latent categories, built from their substring information. This interpretability is very important: opaque and black-box machine learning models have limited adoption in real-world data-science applications. Often, practitioners resort to manual data cleaning to regain interpretability of the models. Finally, we demonstrate on 17 real-life datasets that our encoding methods improve supervised learning on non curated data without the need for dataset-specific choices. As such, these encodings provide a scalable and automated replacement to data cleaning or feature engineering, and restore the benefits of a low-dimensional categorical encoding, as one-hot encoding.

The paper is organized as follows. Section 2 states the problem and the prior art on creating feature vectors from categorical variables. Section 3 details our two encoding approaches. In section 4, we present our experimental study with an emphasis on interpretation and on statistical learning for 17 datasets with non-curated entries and 7 curated ones. Section 5 discusses these results, after which appendices provide information on the datasets and the experiments to facilitate the reproduction of our findings.

2 Problem setting and prior art

The statistics literature often considers datasets that contain only categorical variables with a low cardinality, as datasets222See for example, the Adult dataset (https://archive.ics.uci.edu/ml/datasets/adult) in the UCI repository [10]. In such settings, the popular one-hot encoding is a suitable solution for supervised learning [1]: it models categories as mutually exclusive and, as categories are known a priory, new categories are not expected to appear in the test set. With enough data, supervised learning can then be used to link each category to a target variable.

2.1 High-cardinality categorical variables

However, in many real-world problems, the number of different string entries in a column is very large, often growing with the number of observations (Figure 1). Consider for instance the Drug Directory dataset333 Product listing data for all unfinished, unapproved drugs. Source: U.S. Food and Drug Administration (FDA). One of the variables is a categorical column with non proprietary names of drugs. As entries in this column have not been normalized, many different entries are likely related: they share a common ingredient such as alcohol (see I(a)). Another example is the Employee Salaries dataset444 Annual salary information for employees of the Montgomery County, MD, U.S.A. Source: https://data.montgomerycountymd.gov/. Here, a relevant variable is the position title of employees. As shown in I(b), here there is also overlap in the different occupations.

High-cardinality categorical variables may arise from variability in their string representations, such as abbreviations, special characters, or typos555A taxonomy of different sources of dirty data can be found on [11], and a formal description of data quality problems is proposed by [12].. Such non-normalized data often contains very rare categories. Yet, these categories tend to have common morphological information. Indeed, the number of unique entries grows less fast with the size of the data than the number of words in natural language (Figure 1). In both examples above, drug names and position titles of employees, there is an implicit taxonomy. Crafting feature-engineering or data-cleaning rules can recover a small number of relevant categories. However, it is time consuming and often needs domain expertise.

Notation

We write sets of elements with capital curly fonts, as $\mathcal{X}$. Elements of a vector space (we consider row vectors) are written in bold $\mathbf{x}$ with the $i$-th entry denoted by $x_{i}$, and matrices are in capital and bold $\mathbf{X}$, with $x_{ij}$ the entry on the $i$-th row and $j$-th column.

Let $C$ be a categorical variable such that $\text{dom}(C)\subseteq\mathcal{S}$, the set of finite length strings. We call categories the elements of $\text{dom}(C)$. Let $s_{i}{\in}\mathcal{S},i{=}1...n$, be the category corresponding to the $i$-th sample of a dataset. For statistical learning, we want to find an encoding function $\text{enc:}\,\mathcal{S}\rightarrow\mathbb{R}^{d}$, such as $\text{enc}(s_{i})=\mathbf{x}_{i}$. We call $\mathbf{x}_{i}$ the feature map of $s_{i}$. Table II contains a summary of the main variables used in the next sections.

2.2 One-hot encoding, limitations and extensions

2.2.1 Shortcomings of one-hot encoding

From a statistical-analysis standpoint, the multiplication of entries with related information is challenging for two reasons. First, it dilutes the information: learning on rare categories is hard. Second, with one-hot encoding, representing these as separate categories creates high-dimension feature vectors. This high dimensionality entails large computational and memory costs; it increases the complexity of the associated learning problem, resulting in a poor statistical estimation [13]. Dimensionality reduction of the one-hot encoded matrix can help with this issue, but at the risk of loosing information.

Encoding all unique entries with orthogonal vectors discards the overlap information visible in the string representations. Also, one-hot encoding cannot assign a feature vector to new categories that may appear in the testing set, even if its representation is close to one in the training set. Heuristics such as assigning the zero vector to new categories, create collisions if more than one new category appears. As a result, one-hot encoding is ill suited to online learning settings: if new categories arrive, the entire encoding of the dataset has to be recomputed and the dimensionality of the feature vector becomes unbounded.

2.2.2 Similarity encoding for string categorical variables

For categorical variables represented by strings, similarity encoding extends one-hot encoding by taking into account a measure of string similarity between pairs of categories [7].

Let $s_{i}{\in}\mathcal{S},i{=}1...n$, the category corresponding to the $i$-th sample of a given training dataset. Given a string similarity $\text{sim}(s_{i},s_{j}){:}\,\mathcal{S}{\times}\mathcal{S}{\rightarrow}[0,1]$, similarity encoding builds a feature map $\mathbf{x}_{i}^{\text{sim}}{\in}\mathbb{R}^{k}$ as:

[TABLE]

where $\{s^{(l)},l{=}1\dots k\}\subseteq\mathcal{S}$ is the set of all unique categories in the train set—or a subset of prototype categories chosen heuristically666 In this work, we use as dimensionality reduction technique the k-means strategy explained in [7].. With the previous definition, one-hot encoding corresponds to taking the discrete string similarity:

[TABLE]

where $\mathbbm{1}[\cdot]$ is the indicator function.

Empirical work on databases with categorical columns containing non-normalized entries showed that similarity encoding with a continuous string similarity brings significant benefits upon one-hot encoding [7]. Indeed, it relates rare categories to similar, more frequent ones. In columns with typos or morphological variants of the same information, a simple string similarity is often enough to capture additional information. Similarity encoding outperforms a bag-of-n-grams representation of the input string, as well as methods that encode high-cardinality categorical variables without capturing information in the strings representations [7], such as target encoding [14] or hash encoding [15].

A variety of string similarities can be considered for similarity encoding, but [7] found that a good performer was a similarity based on n-grams of consecutive characters. This n-gram similarity is based on splitting the two strings to compare in their character n-grams and calculating the Jaccard coefficient between these two sets [16]:

[TABLE]

where $\mathcal{G}(s)$ is the set of consecutive character n-grams for the string $s$. Beyond the use of string similarity, an important aspect of similarity encoding is that it is a prototype method, using as prototypes a subset of the categories in the train set.

2.3 Related solutions for encoding string categories

2.3.1 Bag of n-grams

A simple way to capture morphology in a string is to characterize it by the count of its character or word n-grams. This is sometimes called a bag-of-n-grams characterization of strings. Such representation has been shown to be efficient for spelling correction [16] or for named-entity recognition [17]. Other vectorial representations, such as those created by neural networks, can also capture string similarities [18].

For high-cardinality categorical variables, the number of different n-grams tends to increase with the number of samples. Yet, this number increases slower than in a typical NLP problem (see Figure 2). Indeed, categorical variables have less entropy than free text: they are usually repeated, often have subwords in common, and refer to a particular, more restrictive subject.

Representing strings by character-level n-grams is related to vectorizing text by their tokens or words. Common practice uses term-frequency inverse-document-frequency (tf-idf) reweighting: dividing a token’s count in a sample by its count in the whole document. Dimensionality reduction by a singular value decomposition (SVD) on this matrix leads to a simple topic extraction, latent semantic analysis (LSA) [19]. A related but more scalable solution for dimensionality reduction are random projections, which give low-dimensional approximation of Euclidean distances [20, 21].

2.3.2 Word embeddings

If the string entries are common words, an approach to represent them as vectors is to leverage word embeddings developed in natural language processing [22, 23]. Euclidean similarity of these vectors captures related semantic meaning in words. Multiple words can be represented as a weighted sum of their vectors, or with more complex approaches [24]. To cater for out-of-vocabulary strings, FastText [25] considers subword information of words, i.e., character-level n-grams. Hence, it can encode strings even in the presence of typos. Similarly, Bert [26] uses also a composition of substrings to recover the encoding vector of a sentence. In both cases, word vectors computed on very large corpora are available for download. These have captured fine semantic links between words. However, to analyze a given database, the danger of such approach is that the semantic of categories may differ from that in the pretrained model. These encodings do not adapt to the information specific in the data at hand. Moreover, they cannot be trained directly on the categorical variables for two reasons: categories are typically short strings that do not embed enough context; and the number of samples in some datasets is not enough to properly train these models.

3 Scalable encoding of string categories

We now describe two novel approaches for categorical encoding of string variables. Both are based on the character-level structure of categories. The first approach, that we call min-hash encoding, is inspired by the document indexation literature, and in particular the idea of locality-sensitive hashing (LSH) [9]. LSH gives a fast and stateless way to approximate the Jaccard coefficient between two strings [27]. The second approach is the Gamma-Poisson factorization [28], a matrix factorization technique—originally used in the probabilistic topic modeling literature—that assumes a Poisson distribution on the n-gram counts of categories, with a Gamma prior on the activations. An online algorithm of the matrix factorization allows to scale the method with a linear complexity on the number of samples. Both approaches capture the morphological similarity of categories in a reduced dimensionality.

3.1 Min-hash encoding

3.1.1 Background: min-hash

Locality-sensitive hashing (LSH) [9] has been extensively used for approximate nearest neighbor search for learning [29, 30] or as an efficient way of finding similar objects (documents, pictures, etc.) [31] in high-dimensional settings. One of the most famous functions in the LSH family is the min-hash function [27, 32], originally designed to retrieve similar documents in terms of the Jaccard coefficient of the word counts of documents (see [33], chapter 3, for a primer). While min-hash is a classic tool for its collisions properties, as with nearest neighbors, we study it here as encoder for general machine-learning models.

Let $\mathcal{X}^{\star}$ be a totally ordered set and $\pi$ a random permutation of its order. For any non-empty $\mathcal{X}\subseteq\mathcal{X}^{\star}$ with finite cardinality, the min-hash function $Z(\mathcal{X})$ can be defined as:

[TABLE]

Note that $Z(\mathcal{X})$ can be also seen as a random variable. As shown in [27], for any $\mathcal{X},\mathcal{Y}{\subseteq}\mathcal{X}^{\star}$, the min-hash function has the following property:

[TABLE]

Where $J$ is the Jaccard coefficient between the two sets. For a controlled approximation, several random permutations can be taken, which defines a min-hash signature. For $d$ permutations $\pi_{j}$ drawn i.i.d., Equation 5 leads to:

[TABLE]

where $\mathcal{B}$ denotes the Binomial distribution. Dividing the above quantity by $d$ thus gives a consistent estimate of the Jaccard coefficient $J(\mathcal{X},\mathcal{Y})$777 Variations of the min-hash algorithm, as the min-max hash [34] can reduce the variance of the Jaccard similarity approximation. .

Without loss of generality, we can consider the case of $\mathcal{X}^{\star}$ being equal to the real interval $[0,1]$, so now for any $x\in[0,1]$, $\pi_{j}(x)\sim\mathcal{U}(0,1)$.

Proposition 3.1**.**

Marginal distribution.* If $\pi(x)\sim\mathcal{U}(0,1)$, and $\mathcal{X}{\subset}[0,1]$ such that $|\mathcal{X}|{=}k$, then $Z(\mathcal{X})\sim\text{Dir}(k,1)$.*

Proof.

It comes directly from considering that:

$\mathbb{P}(Z(\mathcal{X}){\leq}z)=$ $1-\mathbb{P}(Z(\mathcal{X}){>}z)=1-\prod_{i=1}^{k}\mathbb{P}(\pi(x_{i})>z)=1-(1-z)^{k}$. ∎

Now that we know the distribution of the min-hash random variable, we will show how each dimension of a min-hash signature maps inclusion of sets to simple inequalities.

Proposition 3.2**.**

Inclusion.* Let $\mathcal{X},\mathcal{Y}{\subset}[0,1]$ such that $|\mathcal{X}|{=}k_{x}$ and $|\mathcal{Y}|{=}k_{y}$.*

1. (i)

*If $\mathcal{X}\subset\mathcal{Y}$, then $Z(\mathcal{Y})\leq Z(\mathcal{X})$. * 2. (ii)

\mathbb{P}\bigl{(}Z(\mathcal{Y}){\leq}Z(\mathcal{X})\,\big{|}\,\mathcal{X}{\cap}\mathcal{Y}{=}\emptyset\bigr{)}=\frac{k_{y}}{k_{x}+k_{y}}**

Proof.

(i) is trivial and (ii) comes directly from Prop. 3.1:

[TABLE]

∎

At this point, we do not know anything about the case when $\mathcal{X}\not\subseteq\mathcal{Y}$, so for a fixed $Z(\mathcal{X})$, we can not ensure that any set with lower min-hash value has $\mathcal{X}$ as inclusion. The following theorem allows us to define regions in the vector space generated by the min-hash signature that, with high probability, are associated to inclusion rules.

Theorem 3.1**.**

**Identifiability of inclusion rules.

**Let $\mathcal{X},\mathcal{Y}\,{\subset}\,[0,1]$ be two finite sets such that $|\mathcal{X}|{=}k_{x}$ and $|\mathcal{Y}|{=}k_{y}$. $\forall\,\epsilon{>}0$, if $d\,{\geq}\,\lceil{-}\log(\epsilon)/\log(1{+}\frac{k_{x}}{k_{y}})\rceil$, then:

[TABLE]

Proof.

First, notice that:

[TABLE]

Then, defining $\mathcal{Y^{\prime}}\overset{\text{def}}{=}\mathcal{Y}\setminus(\mathcal{X}{\cap}\mathcal{Y})$, with $|\mathcal{Y^{\prime}}|=k_{y}-k$:

[TABLE]

Finally:

[TABLE]

∎

Theorem 3.1 tells us that taking enough random permutations ensures that when $\forall j,Z_{j}(\mathcal{Y}){\leq}Z_{j}(\mathcal{X})$, the probability that $\mathcal{X}\,{\not\subseteq}\,\mathcal{Y}$ is small. This result is very important, as it shows a global property of the min-hash representation when using several random permutations, going beyond the well-known properties of collisions in the min-hash signature. Figure 9 in the Appendix confirms empirically the bound on the dimensionality $d$ and its logarithmic dependence on the desired false positive rate $\epsilon$.

3.1.2 The min-hash encoder

A practical way to build a computationally efficient implementation of min-hash is to use a hash function with different salt numbers instead of random permutations. Indeed, hash functions can be built with suitable i.i.d. random-process properties [32]. Thus, the min-hash function can be constructed as follows:

[TABLE]

where $h_{j}$ is a hash function888 Here we use a 32bit version of the MurmurHash3 function [35]. on $\mathcal{X}^{\star}$ with salt value $j$.

For the specific problem of categorical data, we are interested in a fast approximation of $J(\mathcal{G}(s_{i}),\mathcal{G}(s_{j}))$, where $\mathcal{G}(s)$ is the set of all consecutive character n-grams for the string $s$. We define the min-hash encoder as:

[TABLE]

Considering the hash functions as random processes, Equation 6 implies that this encoder has the following property:

[TABLE]

Proposition 3.2 tells us that the min-hash encoder transforms the inclusion relations of strings into an order relation in the feature space. This is especially relevant for learning tree-based models, as theorem 3.1 shows that by performing a reduced number of splits in the min-hash dimensions, the space can be divided between the elements that contain and do not contain a given substring $s$.

As an example, Figure 3 shows this global property of the min-hash encoder for the case of the employe salaries dataset with $d{=}2$. The substrings Senior, Supply and Technician are all included in the category Senior Supply Technician, and as consequence, the position for this category in the encoding space will be always in the intersection of the bottom-left regions generated by its substrings.

Finally, this encoder is specially suitable for very large scale settings, as it is very fast to compute and completely stateless. A stateless encoding is very useful for distributed computing: different workers can then process data simultaneously without communication. Its drawback is that, as it relies on hashing, the encoding cannot easily be inverted and interpreted in terms of the original string entries.

3.2 Gamma-Poisson factorization

To facilitate interpretation, we now introduce an encoding approach that estimates a decomposition of the string entries in terms of a linear combination of latent categories.

3.2.1 Model

We use a generative model of strings from latent categories. For this, we rely on the Gamma-Poisson model [28], a matrix factorization-technique well-suited to counting statistics. The idea was originally developed for finding low-dimensional representations, known as topics, of documents given their word count representation. As the string entries we consider are much shorter than text documents and can contain typos, we rely on their substring representation: we represent each observation by its count vector of character-level structure of n-grams. Each observation, a string entry described by its count vector $\mathbf{f}\in\mathbb{N}^{m}$, is modeled as a linear combination of $d$ unknown prototypes or topics, $\mathbf{\Lambda}\in\mathbb{R}^{d\times m}$:

[TABLE]

Here, $\mathbf{x}\in\mathbb{R}^{d}$ are the activations that decompose the observation $\mathbf{f}$ in the prototypes $\mathbf{\Lambda}$ in the count space. As we will see later, these prototypes can be seen as latent categories.

Given a training dataset with $n$ samples, the model estimates the unknown prototypes $\mathbf{\Lambda}$ by factorizing the data’s bag-of-n-grams representation $\mathbf{F}\in\mathbb{N}^{n\times m}$, where $m$ is the number of different n-grams in the data:

[TABLE]

As $\mathbf{f}$ is a vector of counts, it is natural to consider a Poisson distribution for each of its elements:

[TABLE]

For a prior on the elements of $\mathbf{x}\in\mathbb{R}^{d}$, we use a Gamma distribution, as it is the conjugate prior of the Poisson distribution, but also because it can foster a soft sparsity:

[TABLE]

where $\boldsymbol{\alpha}$, $\boldsymbol{\beta}$ $\in\mathbb{R}^{d}$ are the shape and scale parameters of the Gamma distribution for each one of the $d$ topics.

3.2.2 Estimation strategy

To fit the model to the input data, we maximize the log-likelihood of the model, written as:

[TABLE]

Maximizing the log-likelihood with respect to the parameters gives:

[TABLE]

As explained in [28], these expressions are analogous to solving the following non-negative matrix factorization (NMF) with the generalized Kullback-Leibler divergence999 In the sense of the NMF literature. See for instance [36]. as loss:

[TABLE]

In other words, the Gamma-Poisson model can be interpreted as a constrained non-negative matrix factorization in which the generalized Kullback-Leibler divergence is minimized between $\mathbf{F}$ and $\mathbf{X}\mathbf{\Lambda}$, subject to a Gamma prior in the distribution of the elements of $\mathbf{X}$. The Gamma prior induces sparsity in the activations $\mathbf{x}$ of the model.

To solve the NMF problem above, [36] proposes the following recurrences:

[TABLE]

As $\mathbf{F}$ is a sparse matrix, the summations above only need to be computed on the non-zero elements of $\mathbf{F}$. This fact considerably decreases the computational cost of the algorithm.

Following [37], we present an online (or streaming) version of the Gamma-Poisson solver (algorithm 1). The algorithm exploits the fact that in the recursion for $\mathbf{\Lambda}$ (eq. 19 and 20), the summations are done with respect to the training samples. Instead of computing the numerator and denominator in the entire training set at each update, one can update them only with mini-batches of data, which considerably decreases the memory usage and time of the computations.

For better computational performance, we adapt the implementation of this solver to the specificities of our problem—factorizing substring counts across entries of a categorical variable. In particular, we take advantage of the repeated entries by saving a dictionary of the activations for each category in the convergence of the previous mini-batches (algorithm 1, line 4) and use them as an initial guess for the same category in a future mini-batch. This is a warm restart and is especially important in the case of categorical variables because for most datasets, the number of unique categories is much lower than the number of samples.

We also set the hyper-parameters of the algorithm and its initialization for optimal convergence. For $\rho$, the discount factor for the previous iterations of the topic matrix $\mathbf{\Lambda}^{(t)}$ (algorithm 1, line 9-10). choosing $\rho{=}0.95$ gives good convergence speed while avoiding instabilities (Figure 10 in the Appendix). With respect to the initialization of the topic matrix $\mathbf{\Lambda}^{(0)}$, a good option is to choose the centroids of a k-means clustering (Figure 11) in a hashed version101010We use the “hashing trick” [15] to construct a feature matrix without building a full vocabulary, as this avoids a pass on the data and creates a low-dimension representation. of the n-gram count matrix $\mathbf{F}$ and then use as initializations the nearest neighbor observations in the original n-gram space. In the case of a streaming setting, the same approach can be used in a subset of the data.

3.2.3 Inferring feature names

An encoding strategy where each dimension can be understood by humans facilitates the interpretation of the full statistical analysis. A straightforward strategy for interpretation of the Gamma Poisson encoder is to describe each encoding dimension by features of the string entries that it captures. For this, one alternative is to track the feature maps corresponding to each input category, and assign labels based on the input categories that activate the most in a given dimensionality. Another option is to apply the same strategy, but for substrings, such as words contained in the input categories. In the experiments, we follow the second approach as a lot of datasets are composed of entries with overlap, hence individual words carry more information for interpretability than the entire strings.

This method is expected to work well if the encodings are sparse and composed only of non-negative values with a meaningful magnitude. The Gamma-Poisson factorization model ensures these properties.