Meta-Learning for Resampling Recommendation Systems

TL;DR

This paper proposes a meta-learning approach to recommend resampling methods for imbalanced classification datasets, aiming to improve classification quality without exhaustive search.

Contribution

It introduces a novel meta-learning framework for selecting resampling techniques based on dataset properties, addressing the resampling selection problem efficiently.

Findings

Meta-learning effectively recommends resampling methods.

Improves classification performance on imbalanced datasets.

Reduces computational cost compared to exhaustive search.

Abstract

One possible approach to tackle the class imbalance in classification tasks is to resample a training dataset, i.e., to drop some of its elements or to synthesize new ones. There exist several widely-used resampling methods. Recent research showed that the choice of resampling method significantly affects the quality of classification, which raises resampling selection problem. Exhaustive search for optimal resampling is time-consuming and hence it is of limited use. In this paper, we describe an alternative approach to the resampling selection. We follow the meta-learning concept to build resampling recommendation systems, i.e., algorithms recommending resampling for datasets on the basis of their properties.

Figures

Equations

$$RA(S)=\frac{Q_{a(S)}-Q_{\text{worst}(S)}}{Q_{\text{best}(S)}-Q_{\text{worst}(S)}}.$$

$$m_r^{\ast \text{minpvalw}}(S)=\underset{m\in M}{\min }{q}_{r,m}^{\text{pvalw}}(S).$$

$$y_{r,m}(S)=\{\begin{array}{ll}1,& \text{mbox}if{q}_{r,m}^{\text{pval}}(S)<\alpha ,\\ 0,& \text{mbox}otherwise.\end{array}$$

$$a(S)=\{\begin{array}{ll}\underset{r,m}{\mathrm{argmax}}{\hat{p}}_{r,m}(f(S)),& \text{mbox}if\underset{r,m}{\max }{\hat{y}}_{r,m}(f(S))=1,\\ (r^0,1.0),& \text{mbox}otherwise.\end{array}$$

$$y_r(S)=\{\begin{array}{ll}1,& \text{mbox}if\underset{m}{\min }{q}_{r,m}^{\text{pval}}(S)<\alpha ,\\ 0,& \text{mbox}otherwise;\end{array}{z}_r(S)=\underset{m}{\mathrm{argmin}}{q}_{r,m}^{\text{pval}}(S).$$

$$r^{\ast }(S)=\{\begin{array}{ll}\underset{r}{\mathrm{argmax}}{\hat{p}}_r(f(S)),& \text{mbox}if\underset{r}{\max }{\hat{y}}_r(f(S))=1,\\ r^0,& \text{mbox}otherwise;\end{array}{m}^{\ast }(S)=\{\begin{array}{ll}{z}_{r^{\ast }}(S),& \text{mbox}if\underset{r}{\max }{\hat{y}}_r(f(S))=1,\\ 1.0,& \text{mbox}otherwise;\end{array}$$

$$R{A}_a(S)=\frac{q_{a(S)}^{\text{mean}}(S)-\underset{r,m}{\min }{q}_{r,m}^{\text{mean}}(S)}{\underset{r,m}{\max }{q}_{r,m}^{\text{mean}}(S)-\underset{r,m}{\min }{q}_{r,m}^{\text{mean}}(S)}.$$

$$AR{A}_a=\frac{1}{|B|}\sum _{S\in B}R{A}_a(S).$$

Full text

Keywords: resampling method selection, resampling recommendation systems, meta-learning, binary classification, imbalanced datasets

1.  Introduction

In this paper, we consider the task of classification on imbalanced datasets. This is a special case of two-class classification task when one class, ‘minor class’, has much less representatives in the available dataset than the other class, ‘major class’. These conditions are of a big interest because many real-world data analysis problems have inherent peculiarities which lead to unavoidable imbalances in available datasets. Examples of such problems include network intrusion detection (Kruegel et al. [1], Burnaev et al. [2]), oil spill detection from satellite images (Kubat et al. [3], Trekin et al. [4]), detection of fraudulent transactions on credit cards (Chan and Stolfo [5]), diagnosis of rare diseases (Rahman and Davis [6]), prediction and localization of failures in technical systems (Tremblay et al. [7], Alestra et al. [8]), etc. These and many other examples have one common significant feature: target events (diseases, failures, etc.) are rare and therefore they generally constitute only a small fraction of available data. Hence an attempt to naturally formulate any of these problems as a binary classification task (target events form one class, no-events form another class) leads to the class imbalance. Note that this effect is unavoidable since it is caused by the nature of the problem.

Moreover, according to He and Garcia [18], the accurate detection of minor class elements is often the main goal in imbalanced problems. In the above-mentioned examples, minor class corresponds to target events whose accurate prediction is crucial for applications. However, standard classification models (Logistic regression, SVM, Decision tree, Nearest neighbors) treat all classes as equally important and thus tend to be biased towards major class in imbalanced problems (see Ertekin et al. [17], King and Zeng [19], Chawla [20], Cieslak and Chawla [21]). This may lead to inaccurate detection of minor class elements with the average quality of prediction being high. E.g., consider a process with events occurring just $1$% of all times. If a classification model always gives a ‘no-event’ answer it is wrong in just $1$% of all cases. The average quality of the classifier is good. But such prediction is useless for minor class detection. Thus, imbalanced classification problems require special treatment.

One possible way (He and Garcia [18]) to increase importance of minor class and deal with peculiarities described above is to resample dataset in order to soften or remove class imbalance. Resampling may include: oversampling, i.e., addition of synthesized elements to minor class; undersampling, i.e., deletion of particular elements from major class; or both. Resampling is convenient and widely used since it allows to tackle imbalanced tasks using standard classification techniques. On the other hand, it requires the selection of the resampling method.

There exist several resampling methods, i.e. algorithms describing which observations to delete and how to generate new ones. Most of them take the resampling amount as an argument, which governs how many observations are added or deleted. Thus, to apply resampling to a classification task, one has to select a method and a resampling amount.

Previous research by Burnaev et al. [32] showed that there is no ‘universal’ choice of resampling method and resampling amount that would improve classification quality for all imbalanced classification tasks. Moreover, the choice of resampling method strongly affects the quality of the final classification. This influence varies from one dataset to another. Therefore, one has to select resampling method and resampling amount properly for every particular task.

The straightforward approach to select the optimal resampling method and amount is to perform the exhaustive search. One tries all considered options, estimates their quality using cross-validation on the train data, and selects the method with the highest quality. It is ideologically simple and reasonable, but it is also time-consuming. Thus exhaustive search is of limited use.

In this work, we explore an alternative approach to resampling selection problem. We develop resampling recommendation systems, that are algorithms which recommend resampling for datasets on the basis of their properties (statistical, metric, etc.). We aim to construct recommendation systems that satisfy two key requirements. Firstly, they have to be significantly faster than cross-validation exhaustive search. Secondly, they have to give better quality (see section 6.4 for details) than trivial resampling selection strategies, such as using the same resampling method for each dataset.

To construct resampling recommendation systems, we follow the meta-learning approach. We analyze experience of applying resampling methods to various imbalanced classification tasks and used it to learn recommendation systems. Namely, we take a big number of various classification tasks on imbalanced datasets and calculate their characteristics. We apply various resampling methods with various resampling amount values to every task, learn classification models on resampled datasets and estimate their quality. Each task is a learning example (meta-example), its characteristics act as its features (meta-features), quality values act as the target variables. We use the set of meta-examples as the training dataset to learn resampling recommendation systems (for details see section 5).

In this paper, we describe construction and evaluation of resampling recommendation systems. The main contributions of the article are:

Developing two natural approaches to build resampling recommendation systems using meta-learning. 2. 2.

Learning recommendation systems on numerous artificial and real-world imbalanced classification tasks.111Full source code and the datasets (used in the experiment) are available online in Public GitHub repository https://github.com/papart/res-recsyst. 3. 3.

Exploring the performance of the recommendations systems. We show that they give satisfactory results while demanding much less time than exhaustive search.

In section 2 we formulate the task of binary classification on imbalanced datasets and give a brief overview of resampling methods. Section 3 summarizes results of experimental resampling methods comparison by Burnaev et al. [32]. Section 4 provides an overview of previous works on recommendation systems. In section 5, we describe two approaches to construct resampling recommendation systems. Finally, in section 6 we provide all the details of the experimental part of our research, evaluate the quality of resampling recommendation systems and discuss the results.

2.  Imbalanced Classification Task

2.1  Notation and Problem Statement

Consider a dataset with $\ell$ elements $S=(X_{i},y_{i})_{i=1}^{\ell}$, where $X_{i}\in\mathbb{R}^{d}$ is an instance of $d$-dimensional feature space and $y_{i}\in\{0,1\}$ is a class label associated with $X_{i}$. Denote $C_{0}(S)=\{(X_{i},y_{i})\in S\mid y_{i}=0\}$ and $C_{1}(S)=\{(X_{i},y_{i})\in S\mid y_{i}=1\}$. Let label [math] correspond to the major class, label $1$ correspond to the minor class, then $|C_{0}(S)|>|C_{1}(S)|$. To measure a degree of class imbalance for a dataset, we introduce an imbalance ratio $I\!R(S)=\frac{|C_{0}(S)|}{|C_{1}(S)|}$. Note that $I\!R(S)\geq 1$ and the higher it is, the stronger imbalance of $S$ is.

The goal is to learn a classifier using training dataset $S$. This is done in two steps. Firstly, dataset $S$ is resampled by resampling method $r$ with resampling multiplier $m>1$: some observations in $S$ are dropped or some new synthetic observations are added to $S$. Resampling method $r$ determines how observations are deleted or how new ones are synthesized. Multiplier $m$ governs resampling amount by setting the resulting imbalance ratio as $\frac{1}{m}I\!R(S)$. Thereby the result of resampling is a dataset $r_{m}(S)$ with $I\!R(r_{m}(S))=\frac{1}{m}I\!R(S)<I\!R(S)$. Secondly, some standard classification model $h$ is learned on $r_{m}(S)$, which gives classifier $h_{r_{m}(S)}:\mathbb{R}^{d}\rightarrow\leavevmode\nobreak\ \{0,1\}$ as a result.

Performance of a classification model with resampling $(r_{m},h)$ is determined by a predefined classifier quality metrics $Q(h_{r_{m}(S_{\textnormal{train}})},S_{\textnormal{test}})$. It takes as input classifier $h_{r_{m}(S_{\textnormal{train}})}$ learned on resampled training dataset, testing dataset $S_{\textnormal{test}}$ and yields higher value for better classification. In order to determine performance of $r_{m}$ and $h$ on the whole dataset $S$ regarding metrics $Q$, we use a standard procedure based on $k$-fold cross-validation Hastie et al. [22] which yields value of $Q$ for each CV-iteration: $Q^{kCV}(h,r_{m},S)\in\mathbb{R}^{k}$. We will consider their arithmetic mean $Q^{kCV}_{\textnormal{av}}(h,r_{m},S)=\overline{Q^{kCV}(h,r_{m},S)}$ as a quality estimate, but single components of $Q^{kCV}$ will be also used.

It is convenient to regard identity transformation of a dataset as a trivial resampling method. We call it “no-resampling” and denote it as $r^{0}$, then, by definition, $r^{0}_{m}(S)=S$ for any multiplier $m$.

2.2  Resampling Methods Overview

Every resampling method $r$ considered in this paper works according to the following scheme.

Takes input: dataset $S$ (as described in section 2.1), resampling multiplier $m>1$ which determines resulting imbalance ratio as $I\!R(r_{m}(S))=\frac{1}{m}\cdot I\!R(S)$ and thereby controls resampling amount, additional parameters (specific for every particular method). 2. 2.

Modifies given dataset by adding synthesized objects to the minor class (oversampling), or by dropping objects from the major class (undersampling), or both. Details depend on the method used. 3. 3.

Outputs resampled dataset $r_{m}(S)$ with $d$ features and new imbalance ratio ${I\!R(r_{m}(S))=\frac{1}{m}\cdot I\!R(S)}$.

In this paper, we consider three most widely used resampling methods: Random Oversampling, Random Undersampling and Synthetic Minority Oversampling Technique (SMOTE).

2.2.1 Random Oversampling

Random oversampling (ROS, also known as bootstrap oversampling, see He and Garcia [18]) takes no additional input parameters. It adds to the minor class new $(m-1)|C_{1}(S)|$ objects. Each of them is drawn from uniform distribution on $C_{1}(S)$.

2.2.2 Random Undersampling

Random Undersampling (RUS, see He and Garcia [18]) takes no additional parameters. It chooses random subset of $C_{0}(S)$ with $\frac{m-1}{m}\cdot|C_{0}(S)|$ elements and drops them from the dataset. All subsets of $C_{0}(S)$ have equal probabilities to be chosen.

2.2.3 SMOTE

Synthetic Minority Oversampling Technique (SMOTE, see Chawla et al. [9]) takes one additional integer parameter $k$ (number of neighbors). It adds to the minor class new synthesized objects, which are constructed in the following way.

Initialize set as empty: $S_{\textnormal{new}}:=\emptyset$ 2. 2.

Repeat the following steps $(m-1)|C_{1}(S)|$ times:

1. (i)

Randomly select one element $X_{i}\in C_{1}(S)$. 2. (ii)

Find $k$ minor class elements which are nearest neighbors of $X_{i}$. Randomly select one of them and denote it by $X_{j}$. 3. (iii)

Select random point $x$ on the segment [$X_{i},X_{j}]$. 4. (iv)

Assign minor class label to the newly generated element $x$ and store it: $S_{\textnormal{new}}:=S_{\textnormal{new}}\cup\{(x,1)\}.$ 3. 3.

Add generated objects to the dataset: $\tilde{S}=S\cup S_{\textnormal{new}}$.

2.2.4 Other Resampling Methods

There exist several other resampling methods: Tomek Link deletion by Kubat and Matwin [13], One-Sided Selection by Kubat and Matwin [13], Evolutionary Undersampling by Garcia and Herrera [14], borderline-SMOTE by Han et al. [15], Neighborhood Cleaning Rule by Laurikkala [16]. There exist also procedures combining resampling and classification in boosting: SMOTEBoost by Chawla et al. [10], RUSBoost by Seiffert et al. [11], EUSBoost by Galar et al. [12]. These methods are not examined in this paper and not considered in further resampling recommendation system construction, but methodology we describe is quite general, so it can be applied to these methods as well.

3.  Influence of Resampling on Classification Accuracy

The influence of resampling on the classification accuracy was explored experimentally by Burnaev et al. [32]. The authors measured accuracy of various classification models on several artificial and real-world imbalanced datasets, which were preliminarily resampled using one of three methods (Random oversampling, Random undersampling and SMOTE) with multiplier $m$ from 1.25 to 10. Analyzing results of these experiments, the authors came to the following conclusions.

Resampling can have both positive and negative effects on the classification quality. The effect strongly depends on the selected resampling method and multiplier. If the method and the multiplier are selected properly, resampling can significantly improve classification accuracy in most cases. Still, in some cases, classification without resampling is the best choice. In addition, impact of resampling on quality depends on the data it is applied to, so there is no method that would guarantee quality improvement for all datasets.

4.  Recommendation Systems Related Work

The results by Burnaev et al. [32] raise the problem of resampling method and multiplier selection. In fact, proper selection of the method and its parameters is crucial for many other types of data analysis tasks. One possible approach to this problem is to use meta-learning to build systems aimed to recommend method and parameters for task solving. In this section, we discuss construction, application and evaluation of meta-learning-based recommendation systems.

4.1  Recommendation System: Motivation

It is a common case when certain type of data analysis tasks (e.g., classification, time series prediction, feature selection) has plenty of methods for its solution. Suppose that some quality metric is given, so solutions obtained by different methods can be compared. The choice of a method may strongly affect the quality of the solution (e.g., for resampling methods it is demonstrated by Burnaev et al. [32]). In that case, it becomes necessary to select the method properly. The straightforward way to do this is to perform exhaustive search. However, this approach is time-consuming, so its use is practically impossible in some cases. Trying to overcome this problem, many researchers explored an alternative approach to method selection recommendation systems.

By a recommendation system for a certain type of data analysis tasks we mean an algorithm which takes as input a task (its description and its training dataset) and gives as output a recommendation on how to solve this task. The recommendation may be detailed in different ways: it may contain only one method which is considered by the system as the best, or provide ranking of methods preferable for the task, or also provide values of parameters for the recommended methods, etc. Recommendation systems aim to fulfill two requirements. They have to provide recommendations of high quality and they have to be computationally fast. These requirements are conflicting, so a researcher has to decide which one is of higher priority or what trade-off between them is preferable.

One possible approach to recommendation system construction is meta-learning. It has been successfully used for various types of tasks (see section 4.5 for examples) and in the paper we also follow this approach. In subsection 4.2 we describe meta-learning concept and in subsection 4.3 we give the template for meta-learning-based recommendation system construction.

4.2  Meta-Learning Concept

Meta-learning is a widely-used approach to recommendation system construction. According to Vilalta et al. [23], its idea is that there is some intrinsic relation between task properties and performance of methods for its solution, so one can try to extract this relation and use it to build a recommendation system. Generally, the extraction of this relation is a challenging problem. There exists many possible approaches. In the case of meta-learning tasks of the same type are considered as learning examples and are called meta-examples, their properties act as features and are called meta-features, quality values of methods applied to each meta-example (quality-variables) are its target variables (target quality variables). That is, one takes a set of tasks of the same type, calculates their meta-features, applies methods to each of them, computes their quality estimates, gathers them in quality-variables and thereby forms a training meta-dataset. This dataset is used to learn a model predicting quality-variables by meta-features and hence representing the required relation.

When the model is learned, it can predict quality-variables for any task, either from the training meta-dataset or an unseen one. After that, we choose a decision function which will convert these predicted values into a recommendation.

The requirements to recommendation system (see section 4.1) in meta-learning terms mean the following. Firstly, the model predicting quality-variables using meta-features should be as accurate as possible. Secondly, meta-features calculation, quality-variables prediction, decision function application have to be computationally cheap.

Note that the meta-learning concept does not presume specific realization. In this paper, we follow a template described in the following subsection 4.3. Further details of resampling recommendation systems we build are described in section 5.

4.3  Recommendation System Template

In this subsection, we describe a template for meta-learning-based recommendation system realization. It is quite general and covers various recommendation systems from previous works (for details see section 4.5). Resampling recommendation systems we built also fit this template. The template is given below.

Decide the type of considered tasks. 2. 2.

Decide the sets of considered methods and parameters. 3. 3.

Decide the form of recommendation. E.g., single method, several methods, or ranking of methods. 4. 4.

Select a quality metric which will be used to estimate performance of each method on every task. 5. 5.

Select a recommendation accuracy metric which will be used to assess the recommendation system on each task. It can be composed from quality of methods recommended by the system, quality of other methods, runtime of methods. Further discussion of this metric is provided in the following subsection 4.4. 6. 6.

Formulate the meta-learning problem.

1. (a)

Define meta-features. We denote meta-features of a task with dataset $S$ as $f(S)$ and number of them as $n_{f}$. 2. (b)

Define quality-variables which aggregate information about performance of methods on the meta-example. We denote a vector of quality-variables of the task with dataset $S$ by $q(S)$ and a number of them by $n_{q}$. 3. (c)

Define target quality-variables, i.e. some of quality-variables which are to be predicted using meta-features. Note that not every quality-variable is supposed to be a target in the meta-learning problem. Some of them can serve as auxiliary variables used to form the targets or to estimate quality of the recommendation. 7. 7.

Construct the training set of meta-examples.

1. (a)

Collect different tasks of the same type. They will compose a training meta-dataset. Denote this bank of tasks by $B=\{S_{1},\ldots,S_{n_{S}}\}.$ 2. (b)

Calculate the meta-features of every training meta-example: $\{f(S_{1}),\ldots,f(S_{n_{S}})\}.$ 3. (c)

Apply each considered method to every task and get values of quality measure. 4. (d)

Calculate the values of quality-variables for each meta-example in the set: $\{q(S_{1}),\ldots,q(S_{n_{S}})\}.$ 8. 8.

Solve the meta-learning problem by fitting some model on the set of training meta-examples. This model is represented as function $p(f(S))$ taking values of meta-features as input and returning predicted target quality-variables. 9. 9.

Define a decision rule converting target quality-variables into a recommendation. We denote it by $\phi$. 10. 10.

Construct recommendation system using the fitted model and the decision rule. We denote it by $a(S)=\phi(p(f(S)))$, where $a$ is a recommendation system.

Application of such system is straightforward: for a new task with a dataset $S$, calculate meta-features $f(S)$, predict target quality-variables $p(f(S))$, apply the decision rule to get a recommendation: $a(S)=\phi(p(f(S)))$.

4.4  Recommendation System Accuracy

Quality metric value of recommended method $a(S)$ on dataset $S$ is a natural estimate of recommendation system’s accuracy on this dataset. However, there are some important aspects to be considered.

Firstly, higher values of this metric do not imply better recommendation. It is possible that all the considered methods perform well on this particular dataset and $a(S)$ provide the lowest quality among them. Therefore quality values of all the considered methods should be taken into account to estimate accuracy of recommendation.

One possible way to implement this idea was suggested by Song et al. [26]. They introduce the Recommendation Accuracy ($R\!A$) metric:

[TABLE]

Here $Q_{a(S)}$ is a quality metric value of recommended method $a(S)$ on dataset $S$, $Q_{\textnormal{best}(S)}$ and $Q_{\textnormal{worst}(S)}$ are the highest and the lowest quality metric values among all considered methods on dataset $S$. Apparantly, $R\!A(S)\in[0,1]$, and the higher value it has, the better the recommendation for dataset $S$ is. Average value of $R\!A(S)$ on all datasets $S\in B$ can be used to estimate overall accuracy of the recommendation system.

Secondly, the estimate of recommendation system’s efficiency should be based not only on accuracy of recommended methods, but also on the runtime. Indeed, if the runtime is ignored, the exhaustive search would be regarded as the best recommendation system. However, as already mentioned, the exhaustive search can take a lot of time, so it does not fulfill the initial purpose of recommendation system.

In most of previous works, researchers took runtime of recommendation systems into account by setting restrictions on their computatitonal complexity. That is, they considered only computationally cheap meta-features and meta-learning models, which automatically results in low runtime of recommendation systems.

4.5  Related Work on Recommendation Systems

Below we give an overview of recommendation systems from previous papers by Wang et. al. [24], Leite and Brazdil [25], Brazdil et al. [30], Song et al. [26], Furdik et al. [27], Ali and Smith [28], Souto et al. [29], Wang et al. [31]. All of them fit into the template described in subsection 4.3, so we go through its steps and show how they are implemented in each work.

The recommendation system by Wang et. al. [24] recommends a ranking of feature selection methods. The systems by Leite and Brazdil [25], Brazdil et al. [30] recommend a ranking of classification models. The systems by Song et al. [26], Furdik et al. [27], Ali and Smith [28] recommend a single classification model. The system built by Souto et al. [29] recommends a ranking of clustering algorithms. The system by Wang et al. [31] recommends single time-series prediction model.

Researchers usually include in meta-features (see Song et al. [26], Furdik et al. [27], Souto et al. [29], Wang et al. [31]), general information about the task and the training dataset, metric, statistical, information-theoretical properties of the dataset, landmarks (fast estimates of quality, see Leite and Brazdil [25], Vilalta et al. [23]).

The exact form of quality-variables and the meta-learning problem was defined in various ways. For example, Furdik et al. [27], Brazdil et al. [30] formulated a regression problem of the classification score prediction. Wang et. al. [24] formed target variable from a score of a feature selection algorithm. Ali and Smith [28] formulated a multi-class classification problem with labels coding the number of the best method.

Apparently, the formulated meta-learning problems could be solved by different means. The most widespread were $k$-nearest-neighbors-based methods considered by Wang et. al. [24], Leite and Brazdil [25], Song et al. [26], Brazdil et al. [30]. Furdik et al. [27] considered linear regression as one of the models, Ali and Smith [28] used rule-based learning algorithm C$5.0$, Souto et al. [29] used SVM to solve the classification problem.

5.  Resampling Recommendation System Construction

In this section, we describe our implementation of the recommendation system template for the resampling selection problem. We propose two approaches to construct resampling recommendation systems. Both of them consider the same set of meta-features and have the same forms of recommendations, so interfaces of resulting recommendation systems are identical. The approaches differ in meta-learning problem formulation, which results in different inner structure of recommendation systems. Therefore we organize the description as follows. In the first few subsections we describe things common for both approaches: form of recommendation, meta-features, some quality-variables. Then, starting from the meta-learning problem formulation, we describe the approaches separately.

5.1  Purpose and Form of Recommendations

We consider binary classification tasks on imbalanced datasets. According to section 2.1, to solve this task, one has to select resampling method $r$, multiplier $m$ and classification model $h$. Let $h$ be fixed. Let sets of considered resampling methods $R=\{r^{0},\ r^{1},\ldots,r^{n_{1}}\}$ (including no-resampling $r^{0}$) and multipliers $M=\{m_{1},\ldots,m_{n_{2}}\}$ be also fixed. We are going to construct a recommendation system $a$. It will take as input a dataset $S$ and recommend method $r\in R$ and multiplier $m\in M$ for resampling of this dataset before applying classification model $h$ on it ($a(S)=(r,\ m)$).

Note that $h$, $M$ and $R$ are not specified yet since the details of recommendation system algorithm do not depend on it (however, they will be specified in the experimental section 6). In other words, we describe an approach which is scalable to different sets of considered classification models, resampling methods and multipliers.

5.2  Classification Quality Metric

Let some classification quality metric $Q$ (see section 2.1) be fixed. We consider $k$-fold cross-validation estimate $Q^{kCV}_{\textnormal{av}}(h,r_{m},S)$ as the quality measure of resampling method $r$ with multiplier $m$ applied to dataset $S$. The process of recommendation system construction is independent of choice of $Q$ and $k$, so they are specified later in section 6.

5.3  Recommendation Accuracy Metric

The value of $Q^{kCV}_{\textnormal{av}}(h,r_{m},S)$ is also used to estimate the quality of a recommendation given by system $a$ for dataset $S$. According to section 4.4, the time of providing a recommendation should be considered in recommendation system quality metric. We achieve this by setting a restriction on the computational complexity. That is, we consider only recommendation systems which require much less computational time than the exhaustive search and we aim to construct one which would give higher value of $Q^{kCV}_{\textnormal{av}}(h,r_{m},S)$ for any given dataset $S$.

5.4  Meta-features

We use the following list of meta-features: number of objects, number of features, objects-features ratio ($|S|/d$), reversed imbalanced ratio ($1/I\!R$, see section 2.1), distance between centers of classes, minimal and maximal absolute eigenvalue of covariance matrix for each of two classes separately, minimal and maximal value of skewness among all features for each class separately, minimal and maximal $p$-value of skewness normality test among all features for each class separately, minimal and maximal value of kurtosis among all features for each class separately, minimal and maximal $p$-value of kurtosis normality test among all features for each class separately. We also use values of these meta-features in the logarithmic scale.

5.5  Quality-variables

For each dataset $S$, resampling method $r\in R$ (including no-resampling), resampling multiplier $m\in M$ we perform $k$-fold cross-validation and obtain $k$-dimensional vector of quality metric values $Q^{kCV}(h,r_{m},S)$ (one value for each cross-validation iteration). Below we describe how its components are organized into quality-variables for each dataset $S$.

The first group of quality-variables is connected with the performance of classification without resampling. There is only one such variable: $q_{0}^{\textnormal{mean}}(S)$, which is an average of components of $Q^{kCV}(h,r^{0},S)$. The second group of quality-variables describe the classification performance with all considered ways of resampling. That is, for each $r\in\{r^{1},\ldots,r^{n_{1}}\}$ and for each $m\in M$ we compute:

• •

$q_{r,m}^{\textnormal{mean}}(S)$, an average of components of $Q^{kCV}(h,r_{m},S)$.

• •

$q_{r,m}^{\textnormal{pval}}(S)$, which is a $p$-value of the t-test with null hypothesis that mean of $Q^{kCV}(h,r_{m},S)$ is not greater than mean of $Q^{kCV}(h,r^{0},S)$. Thus, the lower this $p$-value is, the better the score of resampling $r_{m}$ is (compared to no-resampling).

• •

$q_{r,m}^{\textnormal{pvalw}}(S)=\max\limits_{m^{\prime}\colon|m^{\prime}-m|<\epsilon}q_{r,m^{\prime}}^{\textnormal{pval}}(S)$ for some fixed $\epsilon$. This is the maximum of $p$-values in $\epsilon$-window around $m$.

In the third group, quality-variables describe the performance of each resampling method $r\in\{r^{1},\ldots,r^{n_{1}}\}$ used with the multiplier which is the best in some sense. There are many ways to define which multiplier is the best, but our recommendation systems are based on the one with the lowest maximum of $p$-values in $\epsilon$-window around it:

[TABLE]

Thus for each $r\in\{r^{1},\ldots,r^{n_{1}}\}$ we include:

• •

$q_{r,\textnormal{minpvalw}}^{\textnormal{mean}}(S)=q_{r,m^{*}}^{\textnormal{mean}}(S)$,

• •

$q_{r,\textnormal{minpvalw}}^{\textnormal{pval}}(S)=q_{r,m^{*}}^{\textnormal{pval}}(S)$, where $m^{*}=m_{r}^{*\textnormal{minpvalw}}(S)$.

5.6  Approach No. 1

This approach requires two additional parameters:

• •

$\alpha\in(0,1)$, which is a significance level for the test that no-resampling is better than resampling;

• •

$\epsilon>0$, which is a half-width of window for multiplier (see previous subsection).

Let $\alpha$ and $\epsilon$ be fixed.

In addition, for each $r\in\{r^{1},\ldots,r^{n_{1}}\}$, $m\in M$ one more quality-variable is introduced:

[TABLE]

Apparently, it represents the result of the above-mentioned test: it equals to $1$ if and only if the null-hypothesis is rejected with significance level $\alpha$ (i.e., resampling $(r,m)$ is better than no-resampling on dataset $S$).

For each $r\in\{r^{1},\ldots,r^{n_{1}}\}$, $m\in M$ the following meta-learning problem is formulated. Each meta-example $S$ is assigned to the class [math] or to the class $1$ according to the value of $y_{r,m}(S)$. The problem is to predict class label of meta-example $S$ using only its meta-features $f(S)$ and provide a probability estimate for this prediction. Informally speaking, the meta-learning problem is to predict whether dataset $S$ is worth being resampled with method $r$ and multiplier $m$ or not.

Solving these meta-learning problems, we obtain a classifier for each $r$ and $m$. It outputs $\hat{y}_{r,m}(f(S))$, which denotes the estimated class label for dataset $S$, and $\hat{p}_{r,m}(f(S))$, which denotes the estimated probability that $S$ actually belongs to class $1$. These estimates are used to form a recommendation in the following natural way:

[TABLE]

Thus, if there is some kind of resampling which is significantly better than no-resampling for dataset $S$ than recommend the resampling with the highest probability of being better; otherwise, recommend no-resampling.

5.7  Approach No. 2

This approach requires two additional parameters:

• •

$\alpha\in(0,1)$, which is a significance level for the test that no-resampling is better than resampling;

• •

$\epsilon>0$, which is a half-width of window for multiplier (see subsection 5.5).

Let $\alpha$ and $\epsilon$ be fixed. For each $r\in\{r^{1},\ldots,r^{n_{1}}\}$, two additional quality-variables are introduced:

[TABLE]

Thus, $y_{r}(S)$ indicates whether resampling method $r$ can give a statistically significant quality improvement on dataset $S$ if the multiplier is chosen properly. Variable $z_{r}(S)$ represents the best choice of multiplier for dataset $S$ resampled with method $r$.

For each resampling method $r\in\{r^{1},\ldots,r^{n_{1}}\}$ we formulate two meta-learning problems.

The first problem is to predict $y_{r}(S)$ for dataset $S$ using its meta-features $f(S)$. This is a binary classification task, and it is very similar to the one stated in the previous subsection. Informally, the problem is to determine whether dataset $S$ is worth being resampled using method $r$ granted that multiplier is chosen properly.

The second problem is to predict $z_{r}(S)$ for dataset $S$ using its meta-features $f(S)$. This is a regression task of predicting most appropriate multiplier $m$ for the resampling method $r$ applied to dataset $S$.

Having these problems solved, we obtain a classifier and a regression model for each $r\in\{r^{1},\ldots,r^{n_{1}}\}$. We denote predicted $y_{r}$ as $\hat{y}_{r}(S)$, probability estimate for this prediction as $\hat{p}_{r}(S)$ and predicted $z_{r}$ as $\hat{z}_{r}(S)$. These estimates are used to form a recommendation in the following way: $a(S)=(r^{*},m^{*})$, where

[TABLE]

Informally, if there is some resampling method (with properly chosen multiplier) which can be significantly better than no-resampling, then we recommend this method and the most appropriate multiplier for it. Otherwise, we recommend no-resampling.

6.  Resampling Recommendation System Learning and Evaluation

In this section, we describe the experimental part of our research: testing the performance of the classification models with resampling on various imbalanced datasets, forming the meta-examples set, learning resampling recommendation systems and assesing their quality.222Full source code and the datasets (used in the experiment) are available online in Public GitHub repository https://github.com/papart/res-recsyst.

6.1  Preparation of Meta-examples Set

In order to learn and evaluate resampling recommendation systems described in section 5, we prepared a set of meta-examples according to the general recommendation system template (see section 4.3). Although we have already specified some aspects of meta-examples construction (see, for example, sections 5.4 and 5.5), some other aspects and technical details still remain undefined. In this subsection, we provide a thorough description of meta-examples set preparation.

6.1.1 Datasets

We used two pools of datasets: artificial ($\sim\!1000$ datasets) and real with ($\sim\!100$ datasets).333Real datasets are availaible online at http://sci2s.ugr.es/keel/imbalanced.php and http://homepage.tudelft.nl/n9d04/occ/index.html. Artificial datasets were drawn from a Gaussian mixture distribution. Each of two classes is a Gaussian mixture with not more than $3$ components. Number of features varies from $6$ to $40$, sizes of datasets varies from $200$ to $1000$, $I\!R$ from $0.05$ to $0.35$. For more details on artificial data generation check the repository https://github.com/papart/res-recsyst. Real-world datasets came from different areas: biology, medicine, engineering, sociology. All features are numeric or binary, their number varies from $3$ to $1000$. Dataset size varies from $200$ to $1000$, and $I\!R$ varies from $0.02$ to $0.75$.

6.1.2 Classification Models

We use Decision trees, $k$-Nearest neighbors ($k=5$), and Logistic regression with $\ell_{1}$ regularization as a classification model ($h$). Optimal parameters of classification models were selected by cross-validation.

6.1.3 Resampling Methods and Multipliers

Set $R$ of resampling methods consists of no-resampling, Bootstrap, RUS and SMOTE with $k\in\{1,3,5,7\}$. Values of resampling multiplier $m$ are from $1.25$ to $10.0$ with step $=0.25$.

6.1.4 Classification Quality Evaluation

The area under precision-recall curve is used as a classification quality metric $Q$. We estimate the quality of classification with resampling as it was described in section 2.1 (with number of cross-validation folds $k=20$). Thus, for each dataset $S$, resampling method $r\in R$, resampling multiplier $m\in M$ and classification model $h$ we obtain $k=20$-dimensional vector of quality metric values $Q^{kCV}(h,r_{m},S)$.

6.1.5 Meta-features

For each dataset $S$, we calculate meta-features listed in section 5.4.

6.1.6 Quality-variables

For each dataset $S$ we use quality metric values $Q^{kCV}(h,r_{m},S)$ to calculate quality-variables (see section 5.5). We set half-width of multiplier window $\epsilon$ to $0.75$.

6.2  Learning and Quality Evaluation Process

According to Burnaev et al. [32], the efficiency of a resampling method on a particular task depends on the classification model. Therefore we treat resampling recommendation problem for each classification model $h$ (see 6.1.2) separately.

For each $h$ we construct two recommendation systems according to the approaches described in subsections 5.6 and 5.7. The learning process has been already described in these subsections (except for values of parameters, selected meta-features and models for meta-problem solving; they will be specified in the following subsection 6.3). However, apart from learning recommendation systems, we need to evaluate their quality, which has not been discussed yet.

In order to evaluate quality of resampling recommendation systems on all availiable datasets, we use $k^{\prime}$-fold cross-validation ($k^{\prime}=10$). That is, we randomly split the bank of datasets $B$ into $k^{\prime}$ subsets $B_{1},\ldots,B_{k^{\prime}}$ of roughly equal size. Then, for each $j=\overline{1,k^{\prime}}$ we perform two steps.

Training. We learn a recommendation system $a_{j}$ as described in section 5.6 on all datasets except those from $B_{j}$. Preparation of meta-examples from datasets is described in details in section 6.1. Selected meta-features and models for recommendation system construction are provided in section 6.3. 2. 2.

*Testing. * We evaluate recommendation quality $Q_{a_{j}(S)}(S)$ values on all datasets $S\in B_{j}$.

As a result, for each dataset $S\in B$ we obtain the quality of recommendation achieved by the system. These quality values are used to evaluate overall performance of recommendation systems, see section 6.4 for details.

6.3  Meta-features and Models

In this subsection we specify the detais of the considered methods and approaches.

6.3.1 Rec. System No. 1 for Decision Tree Classifier

• •

Classification model for solution of the meta-learning problem: AdaBoost with Decision Tree as a base classifier and the number of estimators set to 10.

• •

Meta-features: reversed imbalanced ratio, distance between class centers, number of objects, minimal absolute eigenvalue of covariance matrix of the major class, maximal $p$-value of kurtosis normality test among all features in the minor class. Significance level: $\alpha=0.05$.

6.3.2 Rec. System No. 2 for Decision Tree Classifier

• •

Classification model for solution of the meta-learning problem: AdaBoost with Decision Tree as a base classifier and the number of estimators set to 10.

• •

Regression model for solution of the meta-learning problem: AdaBoost with Decision Tree Regressor as a base model and the number of estimators set to 10.

• •

Meta-features: reversed imbalanced ratio, distance between class centers. Significance level: $\alpha=0.05$.

6.3.3 Rec. System No. 1 for $k$-Nearest Neighbors

• •

Classification model for solution of the meta-learning problem: AdaBoost with Decision Tree as a base classifier and the number of estimators set to 10.

• •

Meta-features: reversed imbalanced ratio, distance between class centers, number of objects, minimal absolute eigenvalue of covariance matrix of the major class, maximal $p$-value of kurtosis normality test among all features in the minor class. Significance level: $\alpha=0.05$.

6.3.4 Rec. System No. 2 for $k$-Nearest Neighbors

• •

Classification model for solution of the meta-learning problem: AdaBoost with Decision Tree as a base classifier and the number of estimators set to 10.

• •

Regression model for solution of the meta-learning problem: AdaBoost with Decision Tree Regressor as a base model and the number of estimators set to 10.

• •

Meta-features: reversed imbalanced ratio, distance between class centers. Significance level: $\alpha=0.05$.

6.3.5 Rec. System No. 1 for $\ell_{1}$ Logistic Regression

• •

Classification model for solution of the meta-learning problem: Logistic Regression with $\ell_{1}$ regularization.

• •

Meta-features: reversed imbalanced ratio, distance between class centers, number of objects, minimal absolute eigenvalue of covariance matrix of the major class, minimal and maximal $p$-value of kurtosis normality test among all features in the minor class, minimal and maximal $p$-value of skewness normality test among all features in the minor class. Significance level: $\alpha=0.3$.

6.3.6 Rec. System No. 2 for $\ell_{1}$ Logistic Regression

• •

Classification model for solution of the meta-learning problem: AdaBoost with Decision Tree as a base classifier and the number of estimators set to 10.

• •

Regression model for solution of the meta-learning problem: AdaBoost with Decision Tree Regressor as a base model and the number of estimators set to 10.

• •

Meta-features: reversed imbalanced ratio, distance between class centers. Significance level: $\alpha=0.05$.

6.4  Resampling Recommendation System Quality Assessment

6.4.1 Accuracy Metric

We use Recommendation Accuracy metric (see section 4.4) to estimate accuracy of resampling recommendation system $a$ on dataset $S$:

[TABLE]

We plot the empirical distribution function of $R\!A$ metric to represent its values on all datasets in a convenient way. That is, for each value of the metric $x=R\!A_{a}(S),S\in B$ we calculate share of datasets $y(x)=\frac{1}{|B|}\left|\left\{S\in B\colon R\!A_{a}(S)<x\right\}\right|$ and plot $y(x)$ versus $x$. The distribution function is non-decreasing and its graph is contained within a unit square. The best possible recommendation system gives $R\!A(S)=1$ on each dataset $S$, so its distribution function starts in $(0,0)$, then goes to the point $(1,0)$ and moves upward to the point $(1,1)$. Therefore the closer to the bottom-right corner $(1,0)$ the distribution function graph is located, the better the recommendation system is.

We also provide average value of $R\!A$ metric on all datasets in order to give a value describing overall accuracy of recommendation system $a$, that is,

[TABLE]

6.4.2 Methods to Compare with

It is reasonable to compare resampling recommendation systems with static strategies of resampling selection. We consider the simplest ones: no-resampling; resampling using Bootstrap, RUS or SMOTE (with $k=5$) which provides balanced classes ($I\!R=1$) in the resulting dataset.

Each of these strategies (as well as each resampling method with any fixed multiplier) can be regarded as a trivial recommendation system. Therefore it is possible to apply recommendation system evaluation methodology from section 6.4.1 directly to them. This allows us to compare resampling recommendation system we built with these strategies in the same terms. More specifically, we are going to compare $R\!A$ empirical distribution functions and values of $AR\!A$.

6.5  Results

Performance of systems recommending resampling for usage with Decision tree classification model is shown in Figure 1. Here both recommendation systems outperform other resampling selection strategies on artificial datasets. However, they do not show a distinct quality improvement on real datasets: SMOTE has a slightly higher average $R\!A$, ROS and no-resample show virtually the same performance as the recommendation systems. This significant difference of performance on artificial and real data can be explained by the fact that the pool of artificial datasets is larger and less diverse than the pool of real ones. We suppose that using a richer real data pool would lead to a more accurate meta-learning problem solution and, consequently, higher performance of recommendation systems.

Figures 3 and 3 show performance of recommendation systems for $k$-Nearest neighbors classifier and $\ell_{1}$ Logistic regression, respectively. In these cases, both recommendation systems achieve a significant quality improvement compared to static strategies ROS, RUS and SMOTE on artificial and real data. However, they have essentially the same quality as no-resampling. It turned out that the systems recommend no-resampling for most of datasets because their meta-features did not indicate that any resampling could be beneficial. This result can be considered as a positive one since no-resampling has high $AR\!A$ value and outperforms other static strategies of resampling selection. On the other hand, recommendation systems failed to recognize those few classification tasks which would have been solved better with resampling. We suppose that it is possible to improve detection of such datasets by introducing new meta-features and more accurately fitting of the meta-learning model.

7.  Conclusion

The meta-learning approach has been successfully applied to various method selection tasks in previous works. We managed to use this approach for recommendation of resampling in imbalanced classification tasks, which has not been done previously. We proposed two natural ways to construct resampling recommendation system and implemented them. The systems we built showed good results: in all cases they achieve $AR\!A>0.6$, which is better than random choice of resampling; for some cases they outperform static strategies of resampling selection.

Besides, we can note several directions of further research which can lead to improvements of recommendation systems quality. First, one can try to introduce new meta-features which are capable of taking into account the specific nature of imbalanced classification tasks. Second, meta-features can be selected more carefully. Third, it is possible to find more accurate models for meta-learning problem solving. Finally, one can use a bigger and more diverse dataset to learn a recommendation system.