A Descriptive Study of Variable Discretization and Cost-Sensitive Logistic Regression on Imbalanced Credit Data

TL;DR

This study explores how variable discretization and cost-sensitive logistic regression can improve classification performance on imbalanced credit data, demonstrating their effectiveness and generalizability across domains.

Contribution

It introduces a combined approach of variable discretization and cost-sensitive logistic regression to mitigate bias in imbalanced datasets, with extensive performance evaluation.

Findings

Discretization outperforms cost-sensitive regression in coefficient estimation.

Proper techniques reduce bias and variance in imbalanced classification.

Methods are effective across multiple domains.

Abstract

Training classification models on imbalanced data tends to result in bias towards the majority class. In this paper, we demonstrate how variable discretization and cost-sensitive logistic regression help mitigate this bias on an imbalanced credit scoring dataset, and further show the application of the variable discretization technique on the data from other domains, demonstrating its potential as a generic technique for classifying imbalanced data beyond credit socring. The performance measurements include ROC curves, Area under ROC Curve (AUC), Type I Error, Type II Error, accuracy, and F1 score. The results show that proper variable discretization and cost-sensitive logistic regression with the best class weights can reduce the model bias and/or variance. From the perspective of the algorithm, cost-sensitive logistic regression is beneficial for increasing the value of predictors even if they are not in their optimized forms while maintaining monotonicity. From the perspective of predictors, the variable discretization performs better than cost-sensitive logistic regression, provides more reasonable coefficient estimates for predictors which have nonlinear relationships against their empirical logit, and is robust to penalty weights on misclassifications of events and non-events determined by their apriori proportions.

Figures

Tables

Table 1. Table 1: The Performance of Variable Discretization on Other Datasets.

Dataset	Model	AUC	Type I Error	Type II Error	Accuracy	F1 Score	Probability Cutoff
arrhythmia	Interval	$0.6216$	$46.87\%$	$42.86\%$	$53.33\%$	$0.1127$	$3.64e-11$
	Discretized	$0.9603$	$14.06\%$	$14.28\%$	$85.92\%$	$0.3871$	$2.09e-17$
wine_quality	Interval	$0.7757$	$30.74\%$	$30.91\%$	$69.25\%$	$0.1439$	$0.0317$
	Discretized	$0.8327$	$26.08\%$	$25.45\%$	$73.95\%$	$0.1763$	$0.0289$

Equations

$$\ln L(\beta |y)=\sum y_i\ln ({\pi }_i)+\sum (1-y_i)\ln (1-{\pi }_i)$$

$$\ln L_W(\beta |y)=W_1\sum y_i\ln ({\pi }_i)+W_0\sum (1-y_i)\ln (1-{\pi }_i)$$

$${\pi }_i=\frac{1}{1+\exp -({\beta }_0+{\beta }_1{x}_1+...+{\beta }_n{x}_n)},W_1=\frac{\tau }{\bar{y}},W_0=\frac{1-\tau }{1-\bar{y}}$$

$$IV=\sum _j(p_j-q_j)\ln (p_j/q_j)$$

Full text

††articletype: Application Note

1 Introduction

Class imbalance problems refer to a class of problems related to classifying imbalanced data where many more observations are labeled by the majority class than the minority class [1] [11]. In practice, the minority class is usually the class of interest, such as fraud in the fraud detection problem [31], malignance in the breast cancer diagnosis problem [23], delinquency in the credit scoring problem [3], sinus bradycardia in the arrhythmia analysis [12], and poor quality in the product quality inspection [5].

However, when trained on imbalanced data, most standard statistics and machine learning models are heavily biased towards the majority class (i.e. non-events) and severely misclassify the minority class (i.e. events) [38], caused by their assumptions of equal target class distribution [17] and maximizing overall accuracy [33]. Models with poor event discrimination are less useful and generate costs associated with Type II errors (money, reputation, health, etc.).

To solve these problems more efficiently, researchers and practitioners have made efforts from various perspectives, such as data sampling [22], feature selection [25] [29], cost-sensitive learning [20] [2] [24], ensemble learning [4], and kernel-based learning [8], with the considerations of concrete problem characteristics.

Previous research has not considered variable discretization as a generic technique for class imbalance problems. In this paper, we empirically explore the effects of variable discretization on classifying imbalanced data and compare it with cost-sensitive logistic regression models. Variable discretization and cost-sensitive logistic regression are studied for their high interpretability and computational efficiency. A credit scoring dataset is used in the case study. The goal is to predict the probability of a debtor’s default or delinquency. The proportion of delinquency observations is only 6.68%. We provide a detailed descriptive study on how variable discretization and cost-sensitive logistic regression help mitigate the model bias and/or variance on an imbalanced credit scoring data. The variable discretization technique is further applied on two datasets from other domains (i.e. biology, business) to demonstrate its potential for use in a wide range of fields.

The paper is structured as follows. In Section 2, related work is reviewed. In Section 3, the data is explored and discretized. In Section 4, the models on the credit scoring dataset are developed, evaluated, and compared. In Section 5, the performance of variable discretization is examined on two datasets from other domains. In Section 6, conclusions and future work are discussed.

2 Related Work

A comprehensive review on the foundations, algorithms, and applications of imbalanced learning was conducted by He et al. in 2013 [15]. It summarized the previous research in five categories, including sampling methods, cost-sensitive methods, kernel-based learning methods, active learning methods, and one-class learning methods. It also suggested to evaluate models based on both curve-based measures (e.g. ROC curve, AUC) and single-value measures (e.g. Type I Error, Type II Error, F1 score, G-mean), considering that some traditional performance measures (e.g. accuracy) did not serve as a good indicator of discrimination abilities of models [34]. In an imbalanced credit scoring study by Wang et al., AUC and F-measure (i.e. F1 score) were used as model performance metrics [39].

In 2001, King proposed the weighted log-likelihood function in Eq. 2 for the logistic regression in rare events data. Compared with the standard log-likelihood function in Eq. 1, Class 1 Weight ($W_{1}$) and Class 0 Weight ($W_{0}$) were added to penalize the misclassifications of events and non-events differently. $W_{1}$ and $W_{0}$ were determined by the estimated population proportion of events $\tau$ and the sample proportion of events $\bar{y}$.

[TABLE]

[TABLE]

where

[TABLE]

The weighted logistic regression in Eq. 2 is referred to as class-dependent cost-sensitive logistic regression [28]. Bahnsen et al. proposed a different version of cost-sensitive logistic regression, called example-dependent cost-sensitive logistic regression [2], where each example (i.e. observation) in the log-likelihood function was associated with a user-defined constant misclassification cost weight based on domain knowledge. Deng and Maher proposed determining each observation’s cost weight by Gaussian kernel function [6] [26] [27], resulting in very high computational complexity $O(n^{3})$ and limiting its application on big data.

Different from cost-sensitive logistic regression which has been widely used, the variable discretization method has not been considered for addressing class imbalance problems, although it has been widely used as a domain-specific standard technique in credit scoring. This technique creates more powerful and interpretable predictors from continuous (i.e. interval) data. Dougherty et al. reviewed existing variable discretization methods, compared three of them (i.e. equal width interval, entropy-based, and purity-based) in depth on 16 datasets, and found that the global entropy-based one performed the best on average [10]. For entropy-based discretization methods, the evaluation measures include: class information entropy, Gini, dissimilarity, and the Hellinger measure [21]. For the scoring problem, one commonly used variable discretization method is called the optimal binning, which computes the cutoff points based on conditional inference trees and recursive partitioning [18].

To select powerful discretized variables, one common measurement is information value defined in Eq. 4 [14], where $p_{j}$ is the number of non-events (i.e. non-delinquency) in the level $j$ of the variable divided by the total number of non-events, and $q_{j}$ is the number of events (i.e. delinquency) in the level $j$ of the variable divided by the total number of events. To interpret the information value, the following rule of thumb is proposed [37] [40].

• •

$<0.02$: useless

• •

$0.02$ to $0.1$: weak

• •

$0.1$ to $0.3$: medium

• •

$>0.3$: strong

[TABLE]

3 Data

Demographic and financial information from $150,000$ borrowers is publicly available in a dataset used in a Kaggle 2011 Competition Give Me Some Credit [19]. The characteristics of the individuals in the data are represented by $11$ variables, as shown in Table 3. The goal was to predict whether a client will experience financial distress in the next two years or not, indicated by the dependent variable $SeriousDlqin2yrs$. As shown in Table 3, there are $10,026$ delinquent observations and $139,937$ non-delinquent observations. The proportion of delinquencies is 6$.68\%$.

There are $29,731$ observations with missing values either in the variable $MonthlyIncome$ or $NumberOfDependents$, which is $19.82\%$ of the total. These missing values are treated as follows.

Missing Completely at Random (MCAR) analysis is conducted, and there is no pattern existing in the missing data. Hence, those observations are dropped to ensure the data accuracy and support the model training computation, when building the model with original variables. After dropping missing data, the proportion of delinquencies is $6.95\%$, which is very close to the original data. 2. 2.

When building the model with discretized variables, those observations are kept by grouping the missing values separately into a level of a variable.

3.1 Exploratory Analysis

Because the dependent variable is binary and all independent variables are interval, the empirical logit plot is used to examine the linearity of the relationship between the dependent variable and independent variables. If the relationship is linear, it is reasonable to use the interval form of an independent variable. Otherwise, a transformation is required. Moreover, through the empirical logit plots, we can check the univariate effects, positive or negative.

The empirical logit plot is created in the following steps.

For each interval variable, generate percentile ranks from $1$ to $100$ [35]. 2. 2.

For each rank $i$ of each interval variable, calculate the total number of observations $N_{i}$, the number of delinquency observations $Y_{i}$, and the mean of the interval variable $\bar{x}_{i}$. 3. 3.

For each rank $i$ of each interval variable, compute the empirical logit using the formula $elogit_{i}=\log(\frac{Y_{i}+0.5}{N_{i}-Y_{i}+0.5})$ [9]. 4. 4.

For each interval variable, plot the empirical logit $elogit$ against the mean in each rank $\bar{x}$ and their linear regression line. Each point in the plot represents $N_{i}$ data points from the dataset by their mean. 5. 5.

For each interval variable, plot the empirical logit $elogit$ against the rank $i$ and their linear regression line. Each point in the plot represents $N_{i}$ data points from the dataset by their rank index.

For example, consider the predictor variable $RevolvingUtilizationOfUnsecuredLines$. Percentile ranks can be found in Table 3.1. Ranks $1-8$ are merged together because their respective minimum and maximum points are the same. As shown in Figure 1a, there is a nonlinear relationship between $RevolvingUtilizationOfUnsecuredLines$ and its empirical logit, mainly caused by extreme values. These extreme values in the empiricial logit plot cannot be simply removed, considering they represent several hundred data points in the dataset. However, the relationship between its rank and its empiricial logit is approximately linear as shown in Figure 1b. In this case, its rank, the discretized form of its original interval values, is preferred to be used in the modeling.

3.2 Variable Discretization

Four variable discretization methods (i.e. distance, quantile, Gini, optimal binning) are compared. On the credit scoring dataset, the quantile discretization produces the highest AUC on the test data with the logistic regression model trained on the training data, where the ratio of training data and test data is 70% vs. 30%. Each variable is ranked and discretized into $20$ bins maximally based on the quantile, with the threshold value $20$ selected by the same procedure above.

Information value is used as the measurement of the discrimination power of each individual variable after discretization, as shown in Table 3.2. Note that for some variables, the resulting number of bins is less than 20 because the bins with non-significant differences are merged together. For the variable $MonthlyIncome$, an additional bin has been included to accomodate missing values. By following the rule suggested by Hand et. al [14], the variables with the information value over $0.1$ will be studied.

To prepare the discretized variables for the modeling, they are further transformed by one-hot encoding. A one-hot encoder converts a discretized variable into multiple binary dummy variables with each bin represented by one binary dummy variable [36] [30].

3.3 Datasets from Other Domains

Beyond the credit scoring data, two public datasets from other domains (i.e. biology, business) are collected. They include 206 and 11 interval variables respectively, as shown in Table 3.3. The goal of the arrhythmia data is to predict sinus bradycardia [12], and the goal of the wine_quality data is to predict poor quality [5]. The process illustrated in Sections 3.1 and 3.2 is performed on these two datasets. Among all variable discretization methods, the optimal binning method produces the best performance. The resulting discretized variables will be modeled using logistic regression in Section 5.

4 Modeling

Logistic regression and class-dependent cost-sensitive logistic regression are used as classifiers for their high interpretability. The models are evaluated by $10$-fold cross-validation. The performance measurements include ROC curve, AUC, Type I Error, Type II Error, accuracy, and F1 Score. The mean of AUCs of $10$-fold cross-validation is used to measure the model bias, while the standard deviation of AUCs of $10$-fold cross-validation is used to measure the model variance. They are reasonable measurements, considering that the model bias refers to the error introduced by approximating the true model, and the model variance refers to the amount of the change of the estimated model if using a different training dataset [16].

To evaluate and compare the performance of variable discretization and class-dependent cost-sensitive logistic regression, the following five models are built.

• •

Model $1$: Logistic regression model on all original interval form of independent variables in Table 3.

• •

Model $2$: Logistic regression model on original interval form of variables with the information value over $0.1$ in Table 3.2.

• •

Model $3$: Class-dependent cost-sensitive logistic regression model on the same independent variables in Model $2$. The class weights (i.e. $W_{0}$, $W_{1}$) that produce the highest mean of AUCs of $10$-fold cross-validation are used in the modeling, indicated by the dash line in Figure 2b. The search for the best class weights will be discussed below.

• •

Model $4$: Logistic Regression model on discretized form of independent variables used in Model $2$. The discretized variables are transformed by the one-hot encoder. In total, $48$ binary dummy variables are created.

• •

Model $5$: Class-dependent cost-sensitive logistic regression model on the same discretized independent variables in Model $4$. The class weights (i.e. $W_{0}$, $W_{1}$) that produce the highest mean of AUCs of $10$-fold cross-validation are used in the modeling, indicated by the solid line in Figure 2b.

For the class weights (i.e. $W_{0}$, $W_{1}$) in Model $3$ and Model $5$, they are determined by the population proportion of events $\tau$ and the sample proportion of events $\bar{y}$ in Eq. 3. $\bar{y}$ is known from the data. $\tau$ is typically unknown and hard to obtain accurate estimation [7]. Here $\tau$ is tuned as a hyperparameter from 0 to 0.5. As shown in Figure 2a, as $\tau$ increases, $W_{1}$ increases and $W_{0}$ decreases linearly. Figure 2b shows how the mean of AUCs on the $10$-fold cross-validation changes as $W1$ increases. When modeling on interval variables in Model $3$, the best occurs at $\tau=0.5$, resulting in $W_{1}=7.19$ and $W_{0}=0.54$. The changes of the class weights have minimal influence on the modeling of discretized variables used in Model $5$, implying that good variable discretization is robust to penalty weights determined by proportions of events and non-events. Hence, for Model $5$, we take $W_{1}=1$ and $W_{0}=1$, leading Model $5$ the same as Model $4$. Because of this, we will only compare Model $4$ with other models in the following section.

The ROC curve of each model can be found in Figure 3. Model $1$ and Model $2$ have similar AUCs, indicating that the variables with the information value below $0.1$ provide minimal contribution. The ROC curves of Model $3$ and Model $4$ demonstrate stronger results than Model $2$. Moreover, for Model $4$, the ROC curves on $10$-fold cross-validation are closer to each other, indicating lower model variance. This can be further confirmed by the mean and standard deviation of AUCs on $10$-fold cross-validation in Table 4. Model $4$ produces the highest mean and the lowest standard deviation of AUCs, demonstrating the power of variable discretization.

The estimated coefficients of the models are also examined. As shown in Table 4, Model $2$ and Model $3$ produce different estimates for every independent variable, as well as the sign of the variable $NumberOfTime60-89DaysPastDueNotWorse$. Its sign is negative in Model $2$, while its sign is positive in Model $3$. Its empirical logit plot in Figure 4c shows the positive relationship. Based on its variance inflation factor (VIF) in Table 4, its sign change in Model $2$ is caused by its multicollinearity with the variables $NumberOfTime30-59DaysPastDueNotWorse$ and $NumberOfTimes90DaysLate$. None of them can be dropped in the modeling because of their information values presented in Table 3.2. Model $3$ specificly guarantees a positive estimate, which is consistent with the univariate effect. For other variables, the signs of estimated parameters are consistent with their univariate effect shown in their empirical logit plots in Figures 4a, 4b, and 4d. The estimated parameters of Model $4$ are not presented here because of space limitation. Considering these dummy variables are binary indicators transformed by one-hot encoder, their estimated coefficients are more interpretable.

Further, these models are compared based on Type I Error, Type II Error, accuracy, and F1 score on the test data after splitting the original dataset into training data (70%) and test data (30%), which can be found in Table 4. The probability cutoff is chosen as the intersection point of the specificity plot and sensitivity plot, one of the most frequently used criterion [13] [32]. We have the following findings.

• •

There is no improvement from Model $1$ to Model $2$, indicating that variables with information value below 0.1 provide limited contribution.

• •

Compared with Model $2$, Model $3$ decreases Type I Error by 8.23%, decreases Type II Error by 8.3%, increases accuracy by 8.24%, and increases F1 score by $0.0656$, indicating the contribution of penalizing the misclassifications of events and non-events in different scales by running the class-dependent logistic regression.

• •

Compared with Model $2$, Model $4$ decreases Type I Error by 11.85%, decreases Type II Error by 11.91%, increases accuracy by 11.85%, and increases F1 score by $0.0946$, indicating the contribution of variable discretization.

• •

Compared with Model $3$, Model $4$ decreases Type I Error by 3.62%, decreases Type II Error by 3.61%, increases accuracy by 3.61%, and increases F1 score by $0.0290$, indicating that variable discretization performs better than the inclusion of class-dependent costs in the logistic regression.

5 Application of Variable Discretization in Other Domains

To further examine the power of variable discretization, logistic regression models with original interval variables and discretized variables in the datasets arrhythmia and wine_quality are built and compared. The original datasets are split into training data (70%) and test data (30%). Logistic regression models are trained on the training data and then evaluated on the test data.

Their resulting ROC curves on the test data can be found in Figure 5. For both datasets, the ROC curve by discretized variables moves closer to the upper-left corner than the one by interval variables. The improvement can be further checked by other performance measures (i.e. Type I Error, Type II Error, accuracy, F1 score) in Table 1, where the probability cutoff is chosen as the intersection point of the sensitivity plot and specificity plot. For example, on the dataset arrhythmia, Type I Error decreases by $32.81\%$, Type II Error decreases by $27.58\%$, accuracy increases by $32.59\%$, and F1 score increases by $0.2744$. Note that the probability cutoff on this dataset is very small, but it is reasonable that some estimated probabilities are very close to 0, considering the facts that they are direct outputs of a sigmoid function ranging from 0 to 1 and target classes (i.e. non-event, event) are represented by 0 and 1 in the data.

6 Discussions and Conclusions

To improve the model performance on imbalanced data, efforts have been made from the perspective of the predictors and the modeling algorithm, respectively, in this study. Through the detailed study on the credit scoring dataset, we show that the proper variable discretization and class-dependent cost-sensitive logistic regression with the best class weights help reduce the model bias and/or variance, based on the ROC curves and AUC on $10$-fold cross-validation, Type I Error, Type II Error, accuracy, and F1 score. Moreover, class-dependent cost-sensitive logistic regression is beneficial for increasing the prediction power of predictors during the training phase even if those predictors are not transformed in their best forms and keeping the multivariate effect and univariate effect of predictors consistent.

On the other hand, the logistic regression model with proper discretized variables performs better than class-dependent cost-sensitive logistic regression, provides more reasonable coefficient estimates, and is robust to penalty scales of misclassification costs of events and non-events determined by their proportions. This indicates that we should always discretize the variables showing nonlinear relationships against their empirical logits.

In this study, logistic regression and its variant (i.e. class-dependent cost sensitive logistic regression) are used as classifiers. In the future, we will study the performance of variable discretization with other classifiers such as decision tree, support vector machine, and neural network.