Progressive Boosting for Class Imbalance
Roghayeh Soleymani, Eric Granger, Giorgio Fumera
TL;DR
This paper introduces Progressive Boosting, a new ensemble learning method that adaptively handles class imbalance by gradually incorporating samples, improving accuracy and reducing complexity in skewed data scenarios.
Contribution
The paper proposes a novel ensemble algorithm called Progressive Boosting that incrementally adds uncorrelated sample groups, enhancing robustness to data skew and computational efficiency.
Findings
Outperforms existing techniques in accuracy across various datasets.
Reduces computational complexity compared to traditional Boosting methods.
Effectively handles variable class imbalance levels.
Abstract
Pattern recognition applications often suffer from skewed data distributions between classes, which may vary during operations w.r.t. the design data. Two-class classification systems designed using skewed data tend to recognize the majority class better than the minority class of interest. Several data-level techniques have been proposed to alleviate this issue by up-sampling minority samples or under-sampling majority samples. However, some informative samples may be neglected by random under-sampling and adding synthetic positive samples through up-sampling adds to training complexity. In this paper, a new ensemble learning algorithm called Progressive Boosting (PBoost) is proposed that progressively inserts uncorrelated groups of samples into a Boosting procedure to avoid loss of information while generating a diverse pool of classifiers. Base classifiers in this ensemble are…
Click any figure to enlarge with its caption.
Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Figure 6
Figure 7
Figure 8
Figure 9
Figure 10
Figure 11
Figure 12
Figure 13
Figure 14
Figure 15
Figure 16
Figure 17
Figure 18
Figure 19
Figure 20
Figure 21
Figure 22
Figure 23
Figure 24
Figure 25
Figure 26| 1:50 | 1:50 | 1:20 | |
| 0.2 | 0.1 | 0.2 |
| Abbreviation | Sampling method | Boosting Ensemble | Loss factor | Data |
| Ada: | Resampling with replacement | AdaBoost [9] | Weighted accuracy | Synthetic, Video, KEEL |
| Ada-F: | Resampling with replacement | Modified AdaBoost [9] | Proposed F-measure | Synthetic, Video |
| SMT: | Synthetic minority over-sampling technique (SMOTE) | SMOTEBoost [10] | Weighted accuracy | Synthetic, Video, KEEL |
| SMT-F | Synthetic minority over-sampling technique(SMOTE) | Modified SMOTEBoost [10] | Proposed F-measure | Synthetic, Video |
| RUS: | Random under-sampling with replacement (RUS) | RUSBoost [14] | Weighted accuracy | Synthetic, Video, KEEL |
| RUS-F: | Random under-sampling with replacement (RUS) | Modified RUSBoost [14] | Proposed F-measure | Synthetic, Video |
| RB: | Combination of up-sampling (SMOTE) and under-sampling (RUS) | RB-Boost [16] | Weighted accuracy | Synthetic, Video, KEEL |
| RB-F: | Combination of up-sampling (SMOTE) and under-sampling (RUS) | Modified RB-Boost [16] | Proposed F-measure | Synthetic, Video |
| PRUS: | Random under-sampling without replacement (RUSwR) | Progressive Boosting | Weighted accuracy | Synthetic, Video |
| PRUS-F: | Random under-sampling without replacement (RUSwR) | Progressive Boosting | Proposed F-measure | Synthetic, Video, KEEL |
| PCUS: | Selecting clusters found by k-means | Progressive Boosting | Weighted accuracy | Synthetic, Video |
| PCUS-F: | Selecting clusters found by k-means | Progressive Boosting | Proposed F-measure | Synthetic, Video, KEEL |
| PCUSi: | Selecting ideal clusters generated in synthetic dataset | Progressive Boosting | Weighted accuracy | Synthetic |
| PCUSi-F: | Selecting ideal clusters generated in synthetic dataset | Progressive Boosting | Proposed F-measure | Synthetic |
| PTUS: | Selecting trajectories in video dataset | Progressive Boosting | Weighted accuracy | Video |
| PTUS-F: | Selecting trajectories in video dataset | Progressive Boosting | Proposed F-measure | Video |
| Ensemble | in iteration | Total | in iteration | Total |
| Ada | ||||
| SMT | ||||
| RUS | ||||
| RB | ||||
| PRUS |
| Ensembles | Train Data | (1:50, ) | (1:50, ) | (1:20, ) | |||||||||
| 1:1 | 1:20 | 1:50 | 1:100 | 1:1 | 1:20 | 1:50 | 1:100 | 1:1 | 1:20 | 1:50 | 1:100 | ||
| F2-measure | |||||||||||||
| Ada | 0.98 0.00 | 0.94 0.00 | 0.91 0.00 | 0.86 0.00 | 0.94 0.01 | 0.85 0.01 | 0.83 0.01 | 0.51 0.01 | 0.89 0.01 | 0.88 0.01 | 0.88 0.01 | 0.46 0.00 | |
| Ada-F | 0.96 0.00 | 0.93 0.00 | 0.90 0.00 | 0.87 0.00 | 0.94 0.01 | 0.88 0.01 | 0.85 0.01 | 0.55 0.01 | 0.92 0.01 | 0.91 0.01 | 0.91 0.01 | 0.48 0.00 | |
| SMT | 0.96 0.01 | 0.90 0.00 | 0.84 0.00 | 0.81 0.00 | 0.93 0.01 | 0.86 0.01 | 0.84 0.01 | 0.58 0.01 | 0.92 0.01 | 0.92 0.01 | 0.92 0.01 | 0.56 0.01 | |
| SMT-F | 0.95 0.01 | 0.89 0.00 | 0.84 0.00 | 0.81 0.00 | 0.95 0.01 | 0.85 0.00 | 0.82 0.00 | 0.55 0.01 | 0.97 0.00 | 0.96 0.00 | 0.96 0.00 | 0.57 0.00 | |
| RUS | 0.93 0.00 | 0.89 0.00 | 0.86 0.00 | 0.83 0.00 | 0.85 0.01 | 0.82 0.01 | 0.81 0.01 | 0.64 0.00 | 0.91 0.01 | 0.91 0.01 | 0.91 0.01 | 0.81 0.01 | |
| RUS-F | 0.95 0.01 | 0.92 0.01 | 0.88 0.00 | 0.85 0.00 | 0.90 0.01 | 0.88 0.01 | 0.84 0.01 | 0.68 0.00 | 0.90 0.01 | 0.90 0.01 | 0.90 0.01 | 0.82 0.01 | |
| RB | 0.99 0.00 | 0.94 0.00 | 0.89 0.00 | 0.86 0.00 | 0.99 0.00 | 0.91 0.00 | 0.91 0.00 | 0.61 0.00 | 0.93 0.01 | 0.93 0.01 | 0.93 0.01 | 0.50 0.01 | |
| RB-F | 0.98 0.00 | 0.95 0.00 | 0.90 0.00 | 0.87 0.00 | 0.97 0.01 | 0.90 0.01 | 0.88 0.01 | 0.59 0.00 | 0.95 0.01 | 0.94 0.01 | 0.94 0.01 | 0.57 0.01 | |
| AUPR | |||||||||||||
| Ada | 0.99 0.00 | 0.97 0.00 | 0.93 0.00 | 0.85 0.00 | 1.00 0.00 | 0.88 0.00 | 0.88 0.00 | 0.60 0.00 | 1.00 0.00 | 1.00 0.00 | 1.00 0.00 | 0.58 0.00 | |
| Ada-F | 0.98 0.00 | 0.96 0.00 | 0.93 0.00 | 0.85 0.00 | 1.00 0.00 | 0.88 0.00 | 0.88 0.00 | 0.60 0.00 | 1.00 0.00 | 1.00 0.00 | 1.00 0.00 | 0.59 0.00 | |
| SMT | 1.00 0.00 | 0.90 0.00 | 0.82 0.00 | 0.78 0.00 | 1.00 0.00 | 0.83 0.00 | 0.81 0.00 | 0.60 0.00 | 1.00 0.00 | 0.98 0.00 | 0.97 0.00 | 0.58 0.00 | |
| SMT-F | 1.00 0.00 | 0.90 0.00 | 0.83 0.00 | 0.78 0.00 | 1.00 0.00 | 0.83 0.00 | 0.81 0.00 | 0.60 0.00 | 1.00 0.00 | 0.98 0.00 | 0.97 0.00 | 0.58 0.00 | |
| RUS | 1.00 0.00 | 0.66 0.00 | 0.60 0.00 | 0.57 0.00 | 1.00 0.00 | 0.74 0.00 | 0.66 0.00 | 0.55 0.00 | 1.00 0.00 | 0.86 0.01 | 0.81 0.01 | 0.57 0.00 | |
| RUS-F | 1.00 0.00 | 0.77 0.01 | 0.70 0.01 | 0.66 0.01 | 1.00 0.00 | 0.74 0.00 | 0.67 0.00 | 0.55 0.00 | 1.00 0.00 | 0.88 0.01 | 0.84 0.01 | 0.57 0.00 | |
| RB | 1.00 0.00 | 0.97 0.00 | 0.93 0.00 | 0.87 0.00 | 1.00 0.00 | 0.88 0.00 | 0.88 0.00 | 0.59 0.01 | 1.00 0.00 | 0.99 0.00 | 0.99 0.00 | 0.60 0.00 | |
| RB-F | 1.00 0.00 | 0.98 0.00 | 0.92 0.00 | 0.86 0.00 | 1.00 0.00 | 0.88 0.00 | 0.88 0.00 | 0.63 0.00 | 1.00 0.00 | 0.99 0.00 | 0.99 0.00 | 0.59 0.00 | |
| Ensembles | ||||||||||
| Ada | 0.41 0.01 | 0.36 0.00 | 0.44 0.03 | 0.82 0.02 | 0.86 0.02 | 0.49 0.01 | 0.61 0.01 | 0.77 0.01 | 0.86 0.01 | 0.88 0.01 |
| Ada-F | 0.40 0.01 | 0.50 0.01 | 0.73 0.02 | 0.82 0.02 | 0.89 0.03 | 0.52 0.01 | 0.64 0.01 | 0.80 0.01 | 0.85 0.01 | 0.89 0.01 |
| SMT | 0.36 0.00 | 0.58 0.00 | 0.82 0.00 | 0.93 0.00 | 0.97 0.00 | 0.43 0.01 | 0.64 0.01 | 0.83 0.00 | 0.83 0.01 | 0.90 0.01 |
| SMT-F | 0.36 0.00 | 0.57 0.00 | 0.83 0.00 | 0.93 0.00 | 0.97 0.00 | 0.47 0.01 | 0.60 0.01 | 0.81 0.01 | 0.92 0.00 | 0.92 0.01 |
| RUS | 0.10 0.00 | 0.22 0.00 | 0.48 0.00 | 0.74 0.00 | 0.85 0.00 | 0.53 0.01 | 0.66 0.01 | 0.81 0.01 | 0.88 0.01 | 0.90 0.00 |
| RUS-F | 0.22 0.00 | 0.26 0.01 | 0.48 0.00 | 0.73 0.00 | 0.85 0.00 | 0.56 0.00 | 0.68 0.01 | 0.83 0.00 | 0.88 0.01 | 0.89 0.01 |
| RB | 0.45 0.00 | 0.67 0.00 | 0.88 0.00 | 0.95 0.00 | 0.97 0.00 | 0.47 0.01 | 0.67 0.00 | 0.85 0.00 | 0.94 0.00 | 0.96 0.00 |
| RB-F | 0.43 0.00 | 0.65 0.00 | 0.86 0.00 | 0.95 0.00 | 0.97 0.00 | 0.45 0.01 | 0.70 0.00 | 0.86 0.00 | 0.93 0.00 | 0.96 0.00 |
| Ensembles | Train Data | (1:50, ) | (1:50, ) | (1:20, ) | |||||||||
| 1:1 | 1:20 | 1:50 | 1:100 | 1:1 | 1:20 | 1:50 | 1:100 | 1:1 | 1:20 | 1:50 | 1:100 | ||
| F2-measure | |||||||||||||
| RUS | 0.93 0.00 | 0.89 0.00 | 0.86 0.00 | 0.83 0.00 | 0.85 0.01 | 0.82 0.01 | 0.81 0.01 | 0.64 0.00 | 0.91 0.01 | 0.91 0.01 | 0.91 0.01 | 0.81 0.01 | |
| PRUS | 0.89 0.01 | 0.86 0.00 | 0.84 0.00 | 0.83 0.00 | 0.86 0.01 | 0.83 0.01 | 0.82 0.01 | 0.66 0.01 | 0.96 0.00 | 0.96 0.00 | 0.96 0.00 | 0.84 0.00 | |
| PCUS | 0.83 0.01 | 0.81 0.01 | 0.79 0.01 | 0.77 0.01 | 0.78 0.01 | 0.74 0.01 | 0.74 0.01 | 0.59 0.01 | 0.91 0.00 | 0.91 0.00 | 0.91 0.00 | 0.79 0.01 | |
| PCUSi | 0.91 0.00 | 0.88 0.00 | 0.85 0.00 | 0.83 0.00 | 0.87 0.01 | 0.84 0.01 | 0.83 0.01 | 0.65 0.01 | 0.93 0.00 | 0.93 0.00 | 0.93 0.00 | 0.84 0.00 | |
| AUPR | |||||||||||||
| RUS | 1.00 0.00 | 0.66 0.00 | 0.60 0.00 | 0.57 0.00 | 1.00 0.00 | 0.74 0.00 | 0.66 0.00 | 0.55 0.00 | 1.00 0.00 | 0.86 0.01 | 0.81 0.01 | 0.57 0.00 | |
| PRUS | 1.00 0.00 | 0.95 0.00 | 0.89 0.00 | 0.87 0.00 | 1.00 0.00 | 0.90 0.00 | 0.89 0.00 | 0.61 0.00 | 1.00 0.00 | 1.00 0.00 | 1.00 0.00 | 0.78 0.01 | |
| PCUS | 0.99 0.00 | 0.91 0.00 | 0.85 0.01 | 0.80 0.01 | 1.00 0.00 | 0.85 0.01 | 0.84 0.01 | 0.56 0.01 | 1.00 0.00 | 1.00 0.00 | 0.99 0.00 | 0.77 0.01 | |
| PCUSi | 1.00 0.00 | 0.93 0.00 | 0.87 0.00 | 0.82 0.00 | 1.00 0.00 | 0.88 0.00 | 0.87 0.00 | 0.65 0.00 | 1.00 0.00 | 1.00 0.00 | 0.99 0.00 | 0.82 0.01 | |
| Ensembles | Train Data | (1:50, ) | (1:50, ) | (1:20, ) | |||||||||
| 1:1 | 1:20 | 1:50 | 1:100 | 1:1 | 1:20 | 1:50 | 1:100 | 1:1 | 1:20 | 1:50 | 1:100 | ||
| F2-measure | |||||||||||||
| Ada | 0.98 0.00 | 0.94 0.00 | 0.91 0.00 | 0.86 0.00 | 0.94 0.01 | 0.85 0.01 | 0.83 0.01 | 0.51 0.01 | 0.89 0.01 | 0.88 0.01 | 0.88 0.01 | 0.46 0.00 | |
| SMT | 0.96 0.01 | 0.90 0.00 | 0.84 0.00 | 0.81 0.00 | 0.93 0.01 | 0.86 0.01 | 0.84 0.01 | 0.58 0.01 | 0.92 0.01 | 0.92 0.01 | 0.92 0.01 | 0.56 0.01 | |
| RB | 0.99 0.00 | 0.94 0.00 | 0.89 0.00 | 0.86 0.00 | 0.99 0.00 | 0.91 0.00 | 0.91 0.00 | 0.61 0.00 | 0.93 0.01 | 0.93 0.01 | 0.93 0.01 | 0.50 0.01 | |
| RUS | 0.93 0.00 | 0.89 0.00 | 0.86 0.00 | 0.83 0.00 | 0.85 0.01 | 0.82 0.01 | 0.81 0.01 | 0.64 0.00 | 0.91 0.01 | 0.91 0.01 | 0.91 0.01 | 0.81 0.01 | |
| PRUS-F | 0.89 0.00 | 0.86 0.00 | 0.84 0.00 | 0.82 0.00 | 0.88 0.01 | 0.84 0.01 | 0.84 0.01 | 0.66 0.00 | 0.94 0.01 | 0.94 0.01 | 0.94 0.01 | 0.81 0.01 | |
| PCUS-F | 0.76 0.01 | 0.74 0.01 | 0.72 0.01 | 0.71 0.01 | 0.77 0.01 | 0.73 0.01 | 0.73 0.01 | 0.60 0.01 | 0.92 0.01 | 0.92 0.01 | 0.92 0.01 | 0.79 0.01 | |
| PCUSi-F | 0.99 0.00 | 0.95 0.01 | 0.89 0.00 | 0.86 0.00 | 0.99 0.00 | 0.92 0.01 | 0.91 0.01 | 0.67 0.00 | 0.95 0.01 | 0.95 0.01 | 0.95 0.01 | 0.84 0.01 | |
| AUPR | |||||||||||||
| Ada | 0.99 0.00 | 0.97 0.00 | 0.93 0.00 | 0.85 0.00 | 1.00 0.00 | 0.88 0.00 | 0.88 0.00 | 0.60 0.00 | 1.00 0.00 | 1.00 0.00 | 1.00 0.00 | 0.58 0.00 | |
| SMT | 1.00 0.00 | 0.90 0.00 | 0.82 0.00 | 0.78 0.00 | 1.00 0.00 | 0.83 0.00 | 0.81 0.00 | 0.60 0.00 | 1.00 0.00 | 0.98 0.00 | 0.97 0.00 | 0.58 0.00 | |
| RB | 1.00 0.00 | 0.97 0.00 | 0.93 0.00 | 0.87 0.00 | 1.00 0.00 | 0.88 0.00 | 0.88 0.00 | 0.59 0.01 | 1.00 0.00 | 0.99 0.00 | 0.99 0.00 | 0.60 0.00 | |
| RUS | 1.00 0.00 | 0.66 0.00 | 0.60 0.00 | 0.57 0.00 | 1.00 0.00 | 0.74 0.00 | 0.66 0.00 | 0.55 0.00 | 1.00 0.00 | 0.86 0.01 | 0.81 0.01 | 0.57 0.00 | |
| PRUS-F | 1.00 0.00 | 0.94 0.00 | 0.90 0.00 | 0.87 0.00 | 1.00 0.00 | 0.90 0.00 | 0.90 0.00 | 0.61 0.00 | 1.00 0.00 | 1.00 0.00 | 1.00 0.00 | 0.76 0.01 | |
| PCUS-F | 0.99 0.00 | 0.89 0.01 | 0.83 0.01 | 0.79 0.01 | 1.00 0.00 | 0.86 0.01 | 0.85 0.01 | 0.56 0.01 | 1.00 0.00 | 0.99 0.00 | 0.99 0.00 | 0.78 0.01 | |
| PCUSi-F | 1.00 0.00 | 0.97 0.00 | 0.91 0.00 | 0.87 0.00 | 1.00 0.00 | 0.89 0.00 | 0.88 0.00 | 0.64 0.00 | 1.00 0.00 | 1.00 0.00 | 1.00 0.00 | 0.82 0.00 | |
| Ensembles | Train Data | 1:50 | 1:100 | ||||||
| 1:1 | 1:20 | 1:50 | 1:100 | 1:1 | 1:20 | 1:50 | 1:100 | ||
| Ada | 0.80 0.01 | 0.86 0.01 | 0.78 0.01 | 0.85 0.01 | 0.75 0.01 | 0.83 0.00 | 0.72 0.01 | 0.82 0.00 | |
| Ada-F | 0.81 0.01 | 0.89 0.00 | 0.80 0.01 | 0.87 0.00 | 0.74 0.01 | 0.84 0.00 | 0.72 0.01 | 0.82 0.00 | |
| SMT | 0.93 0.00 | 0.96 0.00 | 0.92 0.00 | 0.96 0.00 | 0.91 0.00 | 0.95 0.00 | 0.89 0.00 | 0.94 0.00 | |
| SMT-F | 0.93 0.00 | 0.98 0.00 | 0.92 0.00 | 0.97 0.00 | 0.91 0.00 | 0.96 0.00 | 0.90 0.00 | 0.95 0.00 | |
| RUS | 0.94 0.00 | 0.97 0.00 | 0.93 0.00 | 0.96 0.00 | 0.88 0.00 | 0.94 0.00 | 0.87 0.00 | 0.93 0.00 | |
| RUS-F | 0.96 0.00 | 0.98 0.00 | 0.94 0.00 | 0.98 0.00 | 0.90 0.00 | 0.96 0.00 | 0.88 0.00 | 0.94 0.00 | |
| RB | 0.95 0.00 | 0.99 0.00 | 0.94 0.00 | 0.98 0.00 | 0.91 0.00 | 0.97 0.00 | 0.89 0.00 | 0.96 0.00 | |
| RB-F | 0.95 0.00 | 0.98 0.00 | 0.93 0.00 | 0.98 0.00 | 0.90 0.00 | 0.97 0.00 | 0.88 0.00 | 0.96 0.00 | |
| Ensembles | Train Data | 1:50 | 1:100 | ||||||
| 1:1 | 1:20 | 1:50 | 1:100 | 1:1 | 1:20 | 1:50 | 1:100 | ||
| F2-measure | |||||||||
| RUS | 0.94 0.00 | 0.97 0.00 | 0.93 0.00 | 0.96 0.00 | 0.88 0.00 | 0.94 0.00 | 0.87 0.00 | 0.93 0.00 | |
| PRUS | 0.95 0.00 | 0.95 0.00 | 0.92 0.00 | 0.97 0.00 | 0.91 0.00 | 0.95 0.00 | 0.94 0.00 | 0.94 0.00 | |
| PCUS | 0.94 0.00 | 0.95 0.00 | 0.93 0.00 | 0.96 0.00 | 0.90 0.00 | 0.94 0.00 | 0.92 0.00 | 0.92 0.00 | |
| PTUS | 0.96 0.00 | 0.96 0.00 | 0.93 0.00 | 0.95 0.00 | 0.92 0.00 | 0.97 0.00 | 0.98 0.00 | 0.95 0.00 | |
| AUPR | |||||||||
| RUS | 1.00 0.00 | 0.99 0.00 | 0.94 0.00 | 0.97 0.00 | 0.84 0.00 | 0.94 0.00 | 0.80 0.00 | 0.92 0.00 | |
| PRUS | 1.00 0.00 | 0.99 0.00 | 0.92 0.00 | 0.96 0.00 | 0.89 0.00 | 0.94 0.00 | 0.85 0.00 | 0.94 0.00 | |
| PCUS | 1.00 0.00 | 0.99 0.00 | 0.93 0.00 | 0.95 0.00 | 0.88 0.00 | 0.93 0.00 | 0.89 0.00 | 0.93 0.00 | |
| PTUS | 1.00 0.00 | 1.00 0.00 | 0.95 0.00 | 0.97 0.00 | 0.90 0.00 | 0.96 0.00 | 0.89 0.00 | 0.97 0.00 | |
| Ensembles | Train Data | 1:50 | 1:100 | ||||||
| 1:1 | 1:20 | 1:50 | 1:100 | 1:1 | 1:20 | 1:50 | 1:100 | ||
| F2-measure | |||||||||
| Ada | 0.80 0.01 | 0.86 0.01 | 0.78 0.01 | 0.85 0.01 | 0.75 0.01 | 0.83 0.00 | 0.72 0.01 | 0.82 0.00 | |
| SMT | 0.93 0.00 | 0.96 0.00 | 0.92 0.00 | 0.96 0.00 | 0.91 0.00 | 0.95 0.00 | 0.89 0.00 | 0.94 0.00 | |
| RUS | 0.94 0.00 | 0.97 0.00 | 0.93 0.00 | 0.96 0.00 | 0.88 0.00 | 0.94 0.00 | 0.87 0.00 | 0.93 0.00 | |
| RB | 0.95 0.00 | 0.99 0.00 | 0.94 0.00 | 0.98 0.00 | 0.91 0.00 | 0.97 0.00 | 0.89 0.00 | 0.96 0.00 | |
| PRUS-F | 0.95 0.00 | 0.99 0.00 | 0.94 0.00 | 0.98 0.00 | 0.90 0.00 | 0.97 0.00 | 0.90 0.00 | 0.94 0.00 | |
| PCUS-F | 0.94 0.00 | 0.98 0.00 | 0.93 0.00 | 0.96 0.00 | 0.90 0.00 | 0.96 0.00 | 0.89 0.00 | 0.93 0.00 | |
| PTUS-F | 0.96 0.00 | 0.99 0.00 | 0.95 0.00 | 0.99 0.00 | 0.92 0.00 | 0.98 0.00 | 0.91 0.00 | 0.96 0.00 | |
| AUPR | |||||||||
| Ada | 0.96 0.00 | 0.97 0.00 | 0.94 0.00 | 0.97 0.00 | 0.91 0.00 | 0.97 0.00 | 0.89 0.00 | 0.96 0.00 | |
| SMT | 0.96 0.00 | 0.97 0.00 | 0.95 0.00 | 0.97 0.00 | 0.94 0.00 | 0.97 0.00 | 0.94 0.00 | 0.97 0.00 | |
| RUS | 1.00 0.00 | 0.99 0.00 | 0.94 0.00 | 0.97 0.00 | 0.84 0.00 | 0.94 0.00 | 0.80 0.00 | 0.92 0.00 | |
| RB | 1.00 0.00 | 1.00 0.00 | 0.98 0.00 | 1.00 0.00 | 0.96 0.00 | 0.99 0.00 | 0.96 0.00 | 0.99 0.00 | |
| PRUS-F | 1.00 0.00 | 0.98 0.00 | 0.96 0.00 | 0.98 0.00 | 0.98 0.00 | 0.97 0.00 | 0.94 0.00 | 0.95 0.00 | |
| PCUS-F | 1.00 0.00 | 0.96 0.01 | 0.95 0.00 | 0.97 0.02 | 0.89 0.01 | 0.94 0.02 | 0.85 0.01 | 0.93 0.02 | |
| PTUS-F | 1.00 0.00 | 1.00 0.00 | 0.98 0.00 | 1.00 0.00 | 0.98 0.00 | 0.99 0.00 | 0.97 0.00 | 0.99 0.00 | |
| F2-measure | AUPR | ||||||||||||
| Data | , | Ada | SMT | RUS | RB | PRUS-F | PCUS-F | Ada | SMT | RUS | RB | PRUS-F | PCUS-F |
| abalone19 | 129.44 | - | 0.16 0.00 | 0.06 0.00 | 0.24 0.00 | 0.31 0.01 | 0.14 0.01 | - | 0.45 0.00 | 0.43 0.00 | 0.21 0.00 | 0.21 0.00 | 0.25 0.01 |
| abalone9-18 | 16.40 | 0.19 0.01 | 0.63 0.01 | 0.35 0.01 | 0.67 0.01 | 0.62 0.01 | 0.49 0.02 | 0.56 0.01 | 0.62 0.01 | 0.45 0.01 | 0.68 0.02 | 0.65 0.01 | 0.53 0.01 |
| ecoli-0-1-3-7-vs-2-6 | 39.14 | 0.23 0.03 | 0.32 0.04 | 0.28 0.00 | 0.93 0.02 | 0.85 0.00 | 0.82 0.04 | 0.70 0.02 | 0.95 0.01 | 0.75 0.00 | 1.00 0.00 | 0.85 0.00 | 0.95 0.01 |
| ecoli4 | 15.80 | 0.76 0.01 | 0.91 0.00 | 0.76 0.00 | 0.95 0.01 | 0.95 0.00 | 0.95 0.00 | 0.94 0.00 | 0.90 0.01 | 0.80 0.01 | 0.98 0.00 | 0.95 0.00 | 0.97 0.00 |
| glass-0-1-6-vs-2 | 10.29 | 0.18 0.01 | 0.79 0.01 | 0.60 0.01 | 0.89 0.00 | 0.90 0.01 | 0.93 0.00 | 0.59 0.01 | 0.78 0.01 | 0.67 0.00 | 0.86 0.00 | 0.96 0.00 | 0.92 0.01 |
| glass-0-1-6-vs-5 | 19.44 | 0.30 0.04 | 0.97 0.01 | 0.17 0.02 | 1.00 0.00 | 0.96 0.00 | 0.96 0.00 | 0.86 0.01 | 1.00 0.00 | 0.54 0.00 | 1.00 0.00 | 0.93 0.00 | 1.00 0.00 |
| glass2 | 11.59 | 0.12 0.01 | 0.75 0.01 | 0.58 0.00 | 0.83 0.00 | 0.91 0.00 | 0.89 0.01 | 0.62 0.01 | 0.72 0.01 | 0.67 0.00 | 0.75 0.01 | 0.92 0.01 | 0.88 0.01 |
| glass4 | 15.46 | 0.42 0.03 | 0.87 0.01 | 0.21 0.02 | 1.00 0.00 | 0.97 0.00 | 0.97 0.00 | 0.88 0.01 | 0.97 0.00 | 0.57 0.01 | 1.00 0.00 | 0.99 0.00 | 1.00 0.00 |
| glass5 | 22.78 | 0.39 0.03 | 0.96 0.01 | 0.08 0.01 | 1.00 0.00 | 0.96 0.00 | 0.97 0.00 | 0.86 0.01 | 0.99 0.00 | 0.51 0.00 | 1.00 0.00 | 0.98 0.00 | 1.00 0.00 |
| page-blocks-1-3-vs-4 | 15.86 | 0.16 0.01 | 0.22 0.01 | 0.23 0.00 | 0.76 0.01 | 0.54 0.00 | 0.58 0.03 | 0.81 0.00 | 0.81 0.01 | 0.52 0.00 | 1.00 0.00 | 0.95 0.00 | 0.98 0.01 |
| shuttle-c0-vs-c4 | 13.87 | 0.12 0.00 | 0.71 0.02 | 0.24 0.00 | 0.95 0.00 | 0.57 0.00 | 0.71 0.03 | 0.82 0.00 | 0.95 0.00 | 0.85 0.00 | 1.00 0.00 | 0.87 0.02 | 0.87 0.02 |
| shuttle-c2-vs-c4 | 20.50 | 0.32 0.04 | 0.09 0.01 | 0.18 0.00 | 1.00 0.00 | 0.80 0.00 | 0.87 0.02 | 0.77 0.01 | 0.81 0.02 | 0.55 0.00 | 1.00 0.00 | 0.72 0.00 | 0.78 0.01 |
| vowel0 | 9.98 | 0.65 0.01 | 0.98 0.00 | 1.00 0.00 | 1.00 0.00 | 1.00 0.00 | 1.00 0.00 | 0.94 0.00 | 1.00 0.00 | 1.00 0.00 | 1.00 0.00 | 1.00 0.00 | 1.00 0.00 |
| yeast-0-5-6-7-9-vs-4 | 9.35 | 0.41 0.02 | 0.67 0.00 | 0.60 0.00 | 0.66 0.00 | 0.69 0.00 | 0.67 0.01 | 0.64 0.01 | 0.64 0.00 | 0.58 0.00 | 0.72 0.01 | 0.62 0.00 | 0.67 0.01 |
| yeast-1-2-8-9-vs-7 | 30.57 | 0.06 0.01 | 0.38 0.01 | 0.21 0.00 | 0.39 0.00 | 0.45 0.00 | 0.46 0.01 | 0.54 0.01 | 0.49 0.01 | 0.39 0.01 | 0.46 0.01 | 0.42 0.01 | 0.46 0.01 |
| yeast-1-4-5-8-vs-7 | 22.10 | 0.02 0.00 | 0.40 0.00 | 0.24 0.01 | 0.40 0.00 | 0.40 0.00 | 0.46 0.01 | 0.53 0.00 | 0.47 0.00 | 0.34 0.02 | 0.36 0.00 | 0.37 0.00 | 0.47 0.01 |
| yeast-1-vs-7 | 14.30 | 0.24 0.02 | 0.57 0.00 | 0.37 0.01 | 0.56 0.00 | 0.61 0.01 | 0.57 0.01 | 0.58 0.01 | 0.54 0.00 | 0.44 0.02 | 0.57 0.01 | 0.59 0.01 | 0.57 0.01 |
| yeast-2-vs-4 | 9.08 | 0.67 0.02 | 0.85 0.00 | 0.73 0.00 | 0.86 0.00 | 0.84 0.00 | 0.78 0.01 | 0.84 0.01 | 0.82 0.01 | 0.73 0.00 | 0.91 0.01 | 0.87 0.00 | 0.88 0.01 |
| yeast-2-vs-8 | 23.10 | 0.49 0.02 | 0.50 0.00 | 0.43 0.00 | 0.63 0.01 | 0.71 0.00 | 0.69 0.00 | 0.72 0.01 | 0.54 0.00 | 0.53 0.00 | 0.76 0.00 | 0.75 0.00 | 0.75 0.00 |
| yeast4 | 28.10 | 0.10 0.00 | 0.56 0.00 | 0.38 0.00 | 0.60 0.00 | 0.58 0.00 | 0.57 0.00 | 0.44 0.01 | 0.57 0.00 | 0.52 0.00 | 0.55 0.00 | 0.52 0.00 | 0.55 0.01 |
| yeast5 | 32.73 | 0.42 0.02 | 0.81 0.00 | 0.64 0.00 | 0.86 0.00 | 0.83 0.00 | 0.83 0.00 | 0.63 0.00 | 0.79 0.00 | 0.68 0.00 | 0.80 0.00 | 0.80 0.00 | 0.80 0.00 |
| yeast6 | 41.40 | 0.29 0.02 | 0.59 0.01 | 0.38 0.01 | 0.67 0.01 | 0.64 0.01 | 0.66 0.01 | 0.61 0.01 | 0.63 0.01 | 0.54 0.01 | 0.70 0.01 | 0.68 0.01 | 0.69 0.01 |
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsImbalanced Data Classification Techniques · Anomaly Detection Techniques and Applications · Machine Learning and Data Classification
Progressive Boosting for Class Imbalance
Roghayeh Soleymani
Eric Granger
Giorgio Fumera
Laboratoire d’imagerie, de vision et d’intelligence artificielle, École de technologie supérieure
Université du Québec, Montreal, Canada
Pattern Recognition and Applications Group, Dept. of Electrical and Electronic Engineering
University of Cagliari, Cagliari, Italy
Abstract
In practice, pattern recognition applications often suffer from imbalanced data distributions between classes, which may vary during operations w.r.t. the design data. Two-class classification systems designed using imbalanced data tend to recognize the majority (negative) class better, while the class of interest (positive class) often has the smaller number of samples. Several data-level techniques have been proposed to alleviate this issue, where classifier ensembles are designed with balanced data subsets by up-sampling positive samples or under-sampling negative samples. However, some informative samples may be neglected by random under-sampling and adding synthetic positive samples through up-sampling adds to training complexity. In this paper, a new ensemble learning algorithm called Progressive Boosting (PBoost) is proposed that progressively inserts uncorrelated groups of samples into a Boosting procedure to avoid loosing information while generating a diverse pool of classifiers. Base classifiers in this ensemble are generated from one iteration to the next, using subsets from a validation set that grows gradually in size and imbalance. Consequently, PBoost is more robust when the operational data may have unknown and variable levels of skew. In addition, the computation complexity of PBoost is lower than Boosting ensembles in literature that use under-sampling for learning from imbalanced data because not all of the base classifiers are validated on all negative samples. In PBoost algorithm, a new loss factor is proposed to avoid bias of performance towards the negative class. Using this loss factor, the weight update of samples and classifier contribution in final predictions are set based on the ability to recognize both classes. Using the proposed loss factor instead of standard accuracy can avoid biasing performance in any Boosting ensemble. The proposed approach was validated and compared using synthetic data, videos from the Faces In Action dataset that emulates face re-identification applications, and KEEL collection of datasets. Results show that PBoost can outperform state of the art techniques in terms of both accuracy and complexity over different levels of imbalance and overlap between classes.
keywords:
Class Imbalance, Ensemble Learning, Boosting, Face Re-Identification, Video Surveillance.
††journal: Journal of Information Fusion
1 Introduction
Class imbalance is a fundamental issue in many real-world pattern recognition applications found in, e.g., automated video surveillance, fraud detection, intrusion detection in computer and network security, risk management, and medical diagnosis. Imbalance appears in binary classification problems and binarization of multi-class classification problems using one-vs-all strategy when samples from one class are compared against all samples from all other classes [1, 2]. In practice, the level of imbalance observed during operations in unknown a priori and varies over time. This level of skew may differ from what is seen in the design data. Classification algorithms designed using imbalanced data are often biased towards the majority (negative) class, even though the minority class is the (positive) class of interest. The main reason is that learning algorithms are typically designed to optimize the performance in terms of standard accuracy. Consequently, correct classification of negative class becomes their priority due to the abundance of samples for this class.
Several approaches have been proposed in literature to design ensembles of classifiers using imbalanced data [3, 4]. In this paper, these approaches are divided into data-level and algorithm-level approaches. Data-level approaches either up-sample the positive class, under-sample the negative class or combine up-sampling and under-sampling to re-balance data for learning an ensemble of classifiers. Algorithm-level methods create or modify learning algorithms to counter the bias towards the negative class through cost-free techniques or by introducing uneven misclassification costs for the samples from different classes in cost-sensitive approaches.
Ensembles can be designed according to a static or dynamic approach. Static ensembles generate a diverse set of base classifiers a priori, often by re-balancing the training data. The ensembles selection or fusion may be set off-line using validation data, but typically assume a fixed level of imbalance during operations. Dynamic ensembles allow to adapt the selection and fusion of base classifiers during operations based on the estimated level of skew [5, 6]. For example, in [5, 6] authors design base classifiers for a range of different levels of imbalance. Then, they estimate skew level of input data stream and select a suitable fusion function based on that level.
However, the level of imbalance may be difficult to estimate accurately during operations and the diverging selection and fusion function can decrease performance. In contrast, using a static approach, the range of possible imbalance levels can be accounted for during design by training base classifiers on data subsets with different imbalance levels [7].
Most of the ensemble learning methods to handle imbalance in literature are static approaches. Boosting [8, 9] is a common static ensemble method that has been modified in several ways to learn from imbalanced data (see a review by Galar et al. [4]). In data-level Boosting approaches, training data is rebalanced by up-sampling positive class, under-sampling negative class, or using both up-sampling and under-sampling[10, 11, 12, 13, 14, 15, 16]. Up-sampling methods like SMOTEBoost [10] are often more accurate, but they are computationally complex. In contrast, random under-sampling (RUS) [14] is more computationally efficient, but suffers from information loss.
Boosting ensembles may suffer from the bias of performance towards negative class because the loss factor, which guides their learning process, is obtained based on weighted accuracy. In cases of imbalance, weighted accuracy reflects the ability for correct classification of negative samples more than positive ones. This issue can be avoided by adopting a cost-sensitive approach [17, 18, 19], that defines different misclassification costs for different classes and integrates these cost factors into Boosting learning process. The drawback of these cost-sensitive techniques is that they rely on the suitable selection of cost factors which is often estimated by searching a range of possible values. In contrast, cost-free techniques modify learning algorithms by enhancing loss factor calculation without considering cost factors [20, 21].
In this paper, the Progressive Boosting (PBoost) algorithm is proposed to design static classifier ensembles that can maintain a high level of performance over a range of possible levels of imbalance and complexity in the data encountered during operations. In this algorithm, samples from the negative class are regrouped into disjoint partitions, and over iterations, these partitions are gradually accumulated into a temporary design subset. During each Boosting iteration, a new base classifier is trained using negative samples selected randomly from this subset. However, samples from the newly added partition and the important samples from previous iterations have an equally higher probability of being selected. The base classifier is then validated on the whole temporary subset. As with traditional Boosting ensembles, the samples that are misclassified are considered as the most important samples and their weights increase. With the sample selection scheme proposed in this paper, loss of information is considerably reduced, correlation among subsets of negative class is low, and only important samples tend to appear in more than one training subset. Therefore, the diversity and accuracy of Boosting ensembles tend to increase. In addition, to avoid biasing the performance towards the negative class, the proposed Boosting algorithm employs a new loss factor based on the Fβ-measure that is applicable in any Boosting ensemble.
The diverse pool of classifiers generated with PBoost allows to globally model a range of different levels of imbalance and decision bound complexities for the data. Therefore, the static ensembles produced using PBoost are robust to possible variations in data processed during operations because base classifiers are validated on a growing number of negative samples (imbalance level). In addition, the number of samples used per iteration to design (train and validate) a classifier in this ensemble is smaller than Boosting methods in the literature, which translates to a lower computational complexity for design.
The contributions of the proposed PBoost algorithm is summarized as follows:
A sample selection process for design where negative class samples are regrouped into disjoint partitions for training diverse base classifiers to avoid loss of information and bias of performance;
- 2.
A procedure to validate base classifiers on growing number of negative samples to increase robustness to imbalance and decrease computation complexity;
- 3.
A new general loss factor based on the Fβ-measure that is applicable in any Boosting ensemble, to avoid bias of performance towards the negative class.
The PBoost algorithm has been compared to state of the art Boosting ensembles on synthetic, video and KEEL collection datasets in terms of both accuracy and computational complexity.
The rest of the paper is structured as follows. Section 2 contains a review of literature on ensemble learning for class imbalance. In Section 3, the proposed PBoost algorithm is described. The experimental methodology and results are presented in Sections 4 and 5, respectively.
2 Boosting Ensemble Learning for Class Imbalance
Learning from imbalanced data has been addressed in literature through data-level, algorithm-level, and cost-sensitive techniques. Ensemble learning methods exploit one or a combination of aforementioned techniques [4] to handle imbalance. Classifier ensembles can provide higher accuracy and robustness than a single classifier system by combining diverse classifiers [22]. Boosting is a common static ensemble learning algorithm initiated with AdaBoost [8] and improved in AdaBoost.M1 (for 2-class problems) and AdaBoost.M2 (for multiple-class problems) [9] to effectively promote a weak learner that performs slightly better than random guessing into a stronger ensemble. In AdaBoost.M1 (Algo.1) samples are assigned weights that indicate their importance. These weights guide the learning process such that base classifiers in the ensemble focus on correct classification of more important samples as the learning iterations proceed. Samples that are misclassified in each iteration gain more importance for the next iteration and more accurate base classifiers gain higher contribution in final decision. These weights are used directly or for re-sampling training data, depending on the type of the base classifier being used. When the base classifier is from a type that is not designed to incorporate sample weights in its learning process (like SVMs), training data is re-sampled according to the weights of the samples. This case is considered here to explain the Boosting procedure.
Let’s consider a two-class problem with labelled training samples where that contains positive samples and negative samples. All samples in the dataset are initially associated with the same weight , . Then, a new training subset is re-sampled into with to trained classifier . This classifier is tested on all training samples () and a loss factor () is calculated as the sum of the weights of misclassified samples:
[TABLE]
where is the label associated with by . If the classifier is too weak (), the classifier is discarded and training set is re-sampled to train another classifier. The loss factor is then used to define a weight update factor:
[TABLE]
The weights of the samples are then updated as:
[TABLE]
Weight vector is normalized such that the weights of the misclassified samples (more important samples) increase exponentially while the weights of the correctly classified samples decrease. is also used to determine the contribution of the classifier in final predictions (Equation 4) so that more accurate classifiers play more important role in identifying the class of the input sample. This process is repeated for a predefined number of times to design classifiers. Considering as the output of (either a classification score or a label) for an input sample , final prediction of the ensemble is obtained from:
[TABLE]
Analogous to most learning algorithms, AdaBoost is not effective to learn from imbalanced data for two reasons. Negative samples are the majority and when training data is re-sampled in line 2.i of AdaBoost (see Algo. 1), they contribute more in . Therefore, is trained biased to correct classification of this class. After that, when is tested on , loss factor in line 2.iv is calculated as a weighted error rate of classification. Again, negative samples contribute more in loss factor calculation and the weight update formula and classifiers contribution in final prediction become biased such that weight of negative samples increases for the next iteration and classifiers that mostly classify negative samples correctly get higher importance in final prediction of the ensemble. A taxonomy of methods in literature that modify AdaBoost to handle imbalance is presented in Figure 1. Based on the issue these approaches address, they are divided to two categories, data-level and algorithm-level methods that are presented in subsections 2.1 and 2.2, respectively.
2.1 Data-Level Methods:
Class imbalance can be handled in Boosting ensembles through up-sampling the positive class, under-sampling the negative class or combination of them. A popular up-sampling Boosting approach is SMOTEBoost [10] that integrates Synthetic Minority Over-sampling Technique (SMOTE) into AdaBoost.M2. SMOTE creates synthetic samples by interpolating each positive sample with its k-nearest neighbours. MSMOTEBoost [11] use modified SMOTE (MSMOTE) by eliminating noisy samples and oversampling only safe samples. Jous-Boost [12] oversample the positive class by duplicating it, instead of creating new samples, and introduce perturbation (jittering) to this data in order to avoid overfitting. DataBoost-IM [13] oversample difficult samples from both classes and integrates it into AdaBoost.M1 .
Up-sampling techniques address the bias of performance in classifiers through balancing class distribution without loss of information. However, up-sampling, in general, increase the number of samples and consequently increase the complexity of learning algorithms, and SMOTE involves additional computations due to interpolating each sample with its k-nearest neighbours to generate synthetic samples.
In under-sampling Boosting category, RUSBoost [14] integrates random under-sampling (RUS) into AdaBoost.M1. RUSBoost is similar to AdaBoost presented in Algo. 1 where in line 2.i of this algorithm, contains all positive samples and a randomly selected subset of negative class, often with a size equal to the positive class. The subsets of negative class selected randomly over iterations of RUSBoost could be highly correlated and the classifiers trained on them can lack in diversity, especially when the skew level of training data is high. The sample selection paradigm in RUSBoost is managed in EUSBoost [15] to create less correlated subsets using evolutionary prototype selection [23].
Some researchers combine SMOTE and RUS in AdaBoost to achieve greater diversity and avoid loss of information as in Random Balance Boosting (RB-Boost) [16]. RB-Boost combines SMOTE and RUS to create training subsets with random and different skew levels in AdaBoost.M1.
Repetition of sampling in Boosting ensembles increase the chance of low correlation between subsets of data that are used for designing base classifiers and therefore maintain diversity among them. However, some potentially informative samples may be overlooked from these subsets in under-sampling process. In partitional approaches [7, 24, 25] bootstraps are selected without replacement either randomly [24], by clustering [25] or based on a prior knowledge from the application (like trajectories in video surveillance applications such as face re-identification [7]). In these ensemble bootstraps are drawn from a set of negative samples that reduces size in each iteration. In other words, after selection of a bootstrap in each iteration, its samples are eliminated from the main set. In random partitioning of negative samples by Yan et al. [24] the negative data is randomly decomposed into a number of subsets and each subset, combined with the positive samples, is used to train a classifier. Li et al. [25] partition negative data by clustering it using k-means in the feature space and then create an ensemble from the classifiers trained on each negative cluster and the positive samples. The contribution of the classifiers in the ensemble are then weighted based on the distance between the corresponding negative cluster and positive class. In [7], partitioning negative class is done by selecting samples from a set of trajectories that are formed based on the tracking information, as found in several video surveillance applications like face re-identification. In this approach, data from the trajectories are accumulated as the training iteration proceeds and therefore, base classifiers in the ensemble are trained on different imbalance levels to increase robustness of the ensemble to the possible variations in the skew level and complexity of operational data.
In contrast to RUSBoost, these partitional approaches use all negative samples from partitions to design ensembles and avoid loss of information. However, not all samples are informative and using all samples for training may result in unnecessary time and memory complexity. Therefore, enhancing partitional methods with more intelligent sample selection and ensemble learning algorithm (like RUSBoost) can avoid information loss and excessive time complexity at the same time.
2.2 Algorithm-Level Methods:
Using standard error in Boosting ensemble learning algorithms biases their performance towards negative class. In literature this issue is avoided at the algorithm level using two types of techniques; those that employ two different misclassification cost factors, one for positive and another for negative classes and those that handle this issue without the use of cost factors. Cost-sensitive Boosting methods including AdaCost [17], CSB [18] and AdaC [19], embed different misclassification cost factors into loss function or weight update formula of AdaBoost.M2.
Given as the cost factor of sample , in AdaCost [17], two cost adjustment functions are defined for each sample as and and weight update formula is changed to:
[TABLE]
CSB [18] introduce two different cost factors for positive and negative classes as and , respectively.
[TABLE]
In AdaC1, 2, 3 [19] cost factors are embedded into the weight update formula in three different ways. Given , in AdaC1:
[TABLE]
In AdaC2:
[TABLE]
In AdaC3:
[TABLE]
In these cost-sensitive approaches by setting greater than the weights of misclassified samples from positive class increase more than that of the misclassified samples from negative class. In addition, the weights of the classifiers that correctly classify positive class better than the negative class is higher in final decision. Therefore, these cost-sensitive approaches can make up for the usage of standard error rate in Boosting ensembles and allow adapting the performance by selecting proper cost factors based on the application. The drawback of these cost-sensitive approaches is that they require known s that are usually set ad-hoc or by conducting a search in the space of possible costs for a dataset.
Some cost-free approaches have been proposed to deal with the bias of performance caused by using standard error in Boosting ensembles. In RareBoost [20], two different s are defined for positive and negative classes as:
[TABLE]
[TABLE]
where and are the true positive and true negative counts, respectively. Then the weight update formula and final classification prediction are modified as:
[TABLE]
[TABLE]
Kim et al. [21] also define two different s for positive and negative classes as:
[TABLE]
[TABLE]
where and are pseudo errors of classifier in classifying each class. Finally:
[TABLE]
is a multiplier to control the weight of each sample. The problem with this loss factor is that, if there are no misclassified samples in one class or in both classes, is undefined.
Cost-free methods enhance the performance of Boosting ensembles without setting any cost factors and guide the learning process using a more suitable loss factor calculation since the use of weighted standard accuracy, as in original Boosting algorithm, biases the learning process towards correct classification of the negative class. These approaches inspire us to use a more suitable error calculation method in the proposed Boosting algorithm. Therefore, it is relevant to review some of performance measures for imbalanced data classification in the following subsection.
2.3 Performance Measures for Imbalanced Data Classification
The trade-off between true positive rate (TPR) and false positive rate (FPR) for different operating settings can be traced with a Receiver-Operating Characteristic (ROC) curve. ROC curves are widely used to compare classifiers performance. This curve can also be summarized into a global scalar metric; area under the ROC curve (AUC). In addition, for a specific operating setting, G-mean performance measure is defined as the geometrical mean of TPR and TNR (or 1-FPR).
[TABLE]
G-mean gives an equal weight to the efficiency of classifier in correct classification of both classes. ROC space does not adequately reflect the impact of imbalance [26] on performance because big variations in the number of misclassified negative class (FP) results in a small change in FPR, especially if a small increase in TPR can mask it. A suitable alternative for TPR is precision (Pr) that compares the number of misclassified negative samples (FP) to the number of correctly classified positive samples (TP).
[TABLE]
where . It is evident that, as imbalance () increases, any given decrease in FPR results in a higher reduction in Pr. Therefore, precision drops severely when correct classification of positive class is in expense of high misclassification of negative class. For different operating settings, precision-recall (PR) curve depicts the trade off between precision and recall (Re=TPR) when data is imbalanced. Inspired from AUC, area under PR curve (AUPR) can also be used to compare classifiers globally over all operating settings.
For a specific operating setting, precision and recall can be weighted and combined into a local performance metric, the Fβ-measure:
[TABLE]
Although Pr is very sensitive to misclassification of negative samples due to their abundance, with selection of suitable in Fβ-measure, this sensitivity can be controlled. With , this sensitivity reduces and Fβ gives a higher importance to correct classification of positive class. With , this sensitivity increases and correct classification of negative samples becomes the priority.
Another metric that takes imbalance into account is expected cost which is calculated as:
[TABLE]
where is the proportion of positive samples, and and are the misclassification cost of positive and negative classes, respectively. For higher values of (lower values of ) and , EC is more sensitive to increase of FPR rather than FNR. For a specific value of , the sensitivity of EC to each of FNR and FPR can be controlled by tuning and .
Some authors define variants of the existing performance metrics by accounting for [27, 28, 29]. For example, in [28], the authors define a measure of expected accuracy in terms of AUC as , and precision-recall gain (PRG) curve [29] normalize precision and recall in terms of as:
[TABLE]
ROC, PR, and PRG curves are usually produced by varying decision threshold over real valued scores output by classifiers. Consequently, using areas under the curves to generate global metrics or their variants in Boosting ensembles may increase computational cost of these algorithms. Masking the effect of imbalance in performance metrics as done in [27, 28, 29] can misguide the learning process of Boosting ensembles and bias the performance towards negative class. The -measure and expected cost are more suitable performance metrics that take into account correct classification of both classes considering the level of imbalance. However, adjusting the sensitivity of the -measure to the correct classification of the positive and the negative class is easier than expected cost. The reason is that adjusting the -measure involves tuning rather than tuning two factors ( and ) for expected cost. Nevertheless, EC is a more suitable performance metric for cost-sensitive learning algorithms.
In this paper, the -measure, the most frequently used measures for performance evaluation in class imbalance learning, is employed to modify the loss factor calculation in Boosting ensembles, to avoid biasing the performance towards negative class.
3 Progressive Boosting for Learning Ensembles from Imbalanced Data:
The Progressive Boosting (PBoost) learning method is proposed to sustain a high level of performance over a range of imbalance and complexity levels in the data seen during operations. This method follows a static approach, and learns ensembles based on a combination of under-sampling and cost-free adjustment of Boosting ensemble learning.
With the PBoost algorithm, negative class is partitioned into disjoint subsets. These partitions are accumulated into a temporary design set progressively as learning iterations proceed. In each iteration, a subset of this temporary set is used for training a classifier such that the most important samples plus samples from the new partition are given an equally high opportunity to be used in training a base classifier. Loss of information is therefore avoided and ensemble diversity is increased. The trained classifier is then validated on the temporary set that contains all positive samples and only those negative partitions that have already been used in previous training iterations. As the temporary set grows, its imbalance level increases and therefore, the ensemble’s robustness to diverse levels of skew and decision bound complexities during operations is increased. In PBoost, the error of the classifier is determined based on its ability to correctly classify both positive and negative classes. This loss factor plays an important role in determining the contribution of classifiers in final prediction, and in selection criteria of samples for designing the next classifiers.
The progressive Boosting method is presented in Algo. 2 and Figure 2. Its main steps are explained in the following. In the proposed algorithm, the negative samples are first regrouped into disjoint partitions , where , one per classifier in the ensemble (line 1). and the number of negative samples in each partition varies and depends on the partitioning method and the data distribution. There are several possible ways to partition the negative samples into disjoint subsets in literature [30] e.g., prototype-based methods like k-means and GMM algorithms, affinity-based methods like spectral, normalized-cut and sub-space algorithms to represent the negatives, and thus define partitions (number of clusters and association of data to clusters). Two partitioning techniques have been used in literature to partition data to learn ensembles from imbalanced data: Random Under-Sampling without replacement (we call RUSwR in this paper) [24] , and Cluster Under-Sampling (CUS) [25]. In some applications the data is already partitioned, like binarization of multi-class classification problems using one-vs-all strategy. In some applications the data may be grouped based on some contextual or application-based knowledge of data. For example, Trajectory Under-Sampling (TUS) is applicable in video surveillance applications where samples captured for a same individual are regrouped into a trajectory [7]. In the case of random under-sampling without replacement is preselected and takes a fixed random value such that . In the case of CUS and TUS, and depend on the number of samples that are assigned to each partition by the clustering algorithm and the tracker, respectively.
Given a training data set S, one partition is selected in each iteration and added to a temporary set (line 5.ii) which initially contains the positive samples. The same initial weight is assigned to the samples in the new partition creating a weight vector (line 5.i) which is also added to a temporary weight set (line 5.ii). In the next step (line 5.iv), samples from the temporary set are selected through random under-sampling to create a new subset with the weight distribution of . A classifier is trained on (line 5.v). Then it is tested on the whole temporary set that has an imbalance level of (line 5.vi). Therefore, the classifiers in this ensemble are in fact validated on data subsets with a growing level of imbalance and complexity.
After that, a new loss factor is calculated that adapt Boosting algorithm for classifying imbalanced data based on Fβ-measure (line 5.vii) because Fβ-measure is more sensitive to imbalance and at the same time allows us to give more importance to one class than the other.
To calculate the loss factor, we first split the temporary weight vector to two weight matrices for positive and negative classes. The size of is and the size of is , and:
[TABLE]
Then, weighted versions of true positive, false positive, true negative and false negative counts are defined as:
[TABLE]
Based on these values, the accuracy of a classifier is computed in terms of -measure as:
[TABLE]
To measure the error of the classifiers, the corresponding loss factor is defined as:
[TABLE]
The condition in line (v) of AdaBoost.M1 (Algo. 1) means that classifiers in a Boosting ensemble should perform better than random guessing. When Fβ-measure is used as the evaluation metric, the base classifier to beat is the one that predicts everything as positive [29]. Therefore, when the loss factor is calculated using Eq. (33), the accuracy criterion of 0.5 in AdaBoost.M1 should be replaced by (line 5.vii).
After calculation of (line 5.ix) from:
[TABLE]
the weights in the temporary set are updated (line 5.x) as:
[TABLE]
Even though it is desirable to limit the loss of information during under-sampling of data, some samples (like borderline samples) are of more interest than others for training classifiers in the ensemble. In Boosting ensembles, these samples are often detected as misclassified samples because borderline samples play more important role in defining the decision bound and they are more likely to be misclassified. More importance is given to these samples by assigning higher weights to them, so that they have a higher chance to be included in training subset(s). In the proposed PBoost ensemble, after normalization of , its maximum value among negative samples is selected as the initial weight for the next iteration (line 5.xii):
[TABLE]
This value corresponds to the weight of more important misclassified negative samples. Therefore, in each iteration, new samples and misclassified samples from previous iterations have more chance to be included in the training subset. Finally, is used to obtain the final class prediction of the ensemble from (4) (line 6).
PBoost is somewhat inspired from RUSBoost, but differs in three main respects. First, during each iteration, instead of random under-sampling with replacement, most of training negative samples are selected from disjoint partitions. Consequently, repeatedly selection of the same samples over all iterations and information loss is avoided while the diversity increases. Second, instead of validating the classifiers on all samples, the classifiers are validated only on a subset of training set that grows in size and imbalance over iterations. Therefore, robustness to different levels of data imbalance and complexity increases, and the computations complexity of validation step decreases significantly. Third, instead of weighted accuracy, F-measure, an imbalance-compatible performance metric, avoids biasing performance towards negative class.
4 Experimental Methodology
In our experiments, the proposed PBoost ensemble learning method is assessed and compared with AdaBoost.M1 [9], and one state of the art method from each family of the data-level approaches reviewed in Section 2 including SMOTEBoost [10], RUSBoost [14], and RB-Boost [16]. The datasets that are used for the experiments include: (1) A set of synthetic 2D data sets in which the level of skew and overlap between classes are controllable, (2) the Face In Action (FIA) video database [31] that emulates a passport checking scenario in face re-identification application, (3) a set of 21 real-world problems from the KEEL dataset repository [32]. The rest of this section presents the datasets used in the experiments followed by the experimental and evaluation protocols.
4.1 Datasets
4.1.1 Synthetic Dataset
The performance of classification systems may vary on different levels of overlap and skew between classes in both training and test data. Therefore, in our experiments on synthetic data, different synthetic datasets with different overlap and skew levels are generated and used to compare classification systems.
The data is generated to emulate both binarization of a multi-class classification problem when the classification strategy is one versus all and binary classification problems where there is no prior knowledge of optimal partitions. The samples of both positive and negative classes are generated in groups of samples distributed in a normal distribution. The samples from one normal distribution are considered as positive class and all other samples are considered as negative class.
To generate the 2D synthetic data, positive class samples are generated with a normal distribution as , where and indicate the mean and covariance matrix of this distribution, respectively. Then, points are selected randomly in a uniform distribution around . These points (, ) are generated as the mean of Normal distributions (, ) for negative class where . Each normal distribution contains samples and is considered as an ideal cluster of negative class (used for PCUSi).
The mean of these clusters (s) keep a margin distance from . This margin is used to control the level of overlap between positive and negative classes.
For the experiments, we selected the parameter as 0.1 (maximum overlap) and 0.2 (medium overlap). For each overlap level, each normal distribution is randomly divided into two subsets for design and testing. Then the design subsets are divided into 5 folds considering one fold for validation and 4 folds for training. Five replications is possible by alternating the validation fold in each iteration and by reversing the role of design and testing subsets, a total of 10 replications is achieved.
Two settings are considered for the experiments that differ based on the skew level of training data where is set to 1:50 in one setting and to 1:20 in the other. When , only 50 clusters from the negative class are used for training. The objective is to compare different classification algorithms when they are designed on different levels of imbalance. Properties of training data generated with these settings are summarized in Table 1 and examples of training data generated with these settings are presented in Figure 3.
In a similar way, four settings are considered for testing under different imbalance levels () to evaluate the robustness of the classification algorithms over varying skew levels of data during operation. Examples of synthetic test data corresponding to setting is presented in Figure 4.
4.1.2 Face Re-Identification Dataset
Face re-identification is a video surveillance application where individuals in video streams are recognized at different time instants and/or locations over a network of distributed cameras. Non-target faces captured in videos under various challenging conditions are compared to those of the target individual using a video-to-video face recognition system. One important challenge in this application is that the number of faces captured from the target individual (positive class) is typically limited and greatly outnumbered by non-target ones (negative class) [33, 34, 7].
In this classification problem, face captures from each individual may be grouped by the face tracker to trajectories. Given a video stream, an efficient face tracking system, groups face captures from the same individual to a trajectory. A trajectory is Regions Of Interest (ROIs) of a same person regrouped with a face tracker according to a high quality tracking information collected by the tracker that follows the location of the ROIs over consecutive video frames.
FIA video database [31] contains video sequences that emulate a passport checking scenario. The video streams are collected from 221 participants under different capture conditions such as pose, illumination and expression in both indoor and outdoor environments. Videos were collected over three sessions where second and third sessions are three months later than the previous one. The participants are present before 6 cameras for about 5 seconds, resulting in total of 18 video sequences per person.
For experiments in this paper using FIA dataset, only the faces captured with frontal camera in indoor environment is used for both design and testing. ROIs are converted to gray-scale and rescaled to pixels using Viola Jones algorithm [36] from this video. Then, multi-resolution gray-Scale and rotation invariant Local Binary Patterns (LBP) [37] histograms have been extracted as features. The local image texture for LBP has been characterized with 8 neighbours on a 1 radius circle centred on each pixel. Finally, a feature vector with the length of 59 has been obtained for each ROI. Some examples of ROIs from this data set are presented in Figure 5.
For experiments with video data, 10 individuals are randomly selected as targets and 90 individuals are randomly selected as non-targets. In each round of experiment, face patterns of one target individual (a trajectory) is considered as the positive class and 100 individuals (including 9 other target individuals and 90 non-target individuals) are selected as the negative class 111The ROIs of each individual in FIA dataset are already grouped to trajectories.. ROI patterns from each trajectory are divided into 2 sets for design and testing. The design set is divided to 5 folds, and for each round one fold is considered for validation and remaining 4 folds are considered for training. Then the roles of design and testing sets is reversed. Therefore, for each target individual, three independent sets are collected from these face patterns for training, validation and testing. Each set contains one group of samples from the target individual, 9 groups of samples from the remaining target individuals and 90 groups of samples from non-target individuals. Repeating this process for each target individual yields overall experiments for this dataset.
Two imbalance levels ( and 100) and four different imbalance levels are considered for selecting the training and testing negative class for each positive individual, respectively. This is to evaluate the performance of different classification algorithms when they are trained on different imbalance levels, and to evaluate the robustness of the classification algorithms over varying skew levels during operations. When , for each positive individual, only of 100 other individuals are used as the negative class from the training set that was collected for that positive individual. Therefore, when , there are 50 negative individuals in the testing set that were not included in training the classification systems and the skew level of test data is higher than the skew level of training data. When , most of the negative individuals that were used for training do not appear in testing data. When , the maximum imbalance level of testing data is the same as the imbalance level of training data. Therefore, all individuals are seen in both training and testing. However, in this case a high level of imbalance exists in both training and testing stages that makes both learning and classification more difficult. It is worth mentioning that in all settings, the skew level of the validation data is selected to be the same as testing data.
4.1.3 KEEL Collection
KEEL (Knowledge Extraction based on Evolutionary Learning) tool is an open source software that supports data management and a designer of experiments [32]. The dataset collection in KEEL format contains several datasets for binary classification problem with different number of samples, attributes and imbalance levels. In this paper, the first group of this collection 222(http://www.KEEL.es/dataset.php) is used for experiments. In this group, the skew level of datasets ranges between 1:9 and 1:129. The experiments on this collection is done using stratified 2 5 fold cross validation strategy. Therefore, the skew levels of training, validation and testing sets are equal.
4.2 Experimental Protocol
For validation, datasets are selected and generated to consider two possible cases. Synthetic and video datasets are used to evaluate the algorithm when the ideal partitions (or clusters) of negative class are known a priori. The synthetic and video sets are also used for a binary classification problem where no information is available regarding the ideal clusters of data. The KEEL collection datasets are also a case when the data is not partitioned a priori.
We use SVM with RBF kernel [38] as the base classifier where . The kernel parameter is set as the average of the mean minimum distance between any two training samples and the scatter radius of the training samples in the input space [39]. The scatter radius is calculated by selecting the maximum distance between the training samples and a point corresponding to the mean of training samples. We used the LibSVM implementation of [38].
A brief description of the implemented ensembles, their variants and the abbreviations used for them are shown in Table 2. The last column of the table shows the datasets that are used for experiments on these classification systems. The abbreviations assigned to these ensembles are selected based on their sampling techniques and loss factor.
The baseline sampling techniques include Ada (resampling in AdaBoost), SMT (SMOTE in SMOTEBoost), RUS (random under-sampling in RUSBoost), RB (random balance in RB-Boost). For PBoost four partitioning techniques are used for under-sampling the negative class to evaluate the effect of the partitioning technique on the performance of PBoost ensemble: random under sampling without replacement (PRUS) and cluster under-sampling (PCUS) are used as general partitioning techniques for PBoost disregarding the data structure, whether or not the negative class is partitioned a priori. For PCUS, kernel -means is used for clustering negative samples. To select , it is varied over a range of possible values and the value of Dunn index [40] is calculated for each case using a validation set. Finally, the optimal , is selected when Dunn index takes its maximum value. Two cases are considered for PBoost in which the partitions of the negative class are known a priori. The ideal cluster under-sampling (PCUSi) with synthetic datasets and trajectory under-sampling (PTUS) with video dataset.
The loss factor is calculated in 2 ways based on: the traditional technique i.e. weighted accuracy, and the F-measure. To indicate the use of proposed loss factor in the Boosting ensembles in Table 2, the abbreviation is followed by -F. For the use of proposed loss factor calculation with the F-measure, is set as 2 in all experiments because is more suitable for imbalanced data classification when the positive class is the minority class. An experiment is done to evaluate the performance of Boosting ensembles with different values of .
In the experiments with synthetic and video data sets, two different imbalance levels are used for training and four different imbalance levels are used for testing. This is to evaluate the sensitivity of classification systems to the level of imbalance during training and their robustness to possible variations in skew level during operations. In experiments with synthetic data, the overlap level between positive and negative classes are also varied because the issue of imbalance is related to the level of overlap between classes [41].
In experiments with synthetic and video datasets, the size of all Boosting ensembles is set equal to the maximum imbalance level of the data, except from PCUS. The reason for this setting is that the number of ideal clusters and the number of trajectories are both known and equal to the level of skew. In addition, based on a preliminary experiment with D2 on baseline ensembles in Figure 6, it is observed that the size of these ensembles does not have a significant impact on their performance. The performance of these ensembles vary in terms of F2-measure as the ensemble size grows. However, their global performance in terms of AUPR do not change significantly. For PCUS, the size of ensemble is selected equal to the optimal k obtained using Dunn index.
4.3 Evaluation Protocol
Global performance evaluation curves such as ROC and PR, show the trade off between two metrics for different operational settings. For classifiers that output scores or probability estimates, this setting is usually the choice of decision threshold. Area under the curve, shows the global performance of the classifier over a range of possible decision thresholds, where local evaluation metric such as F-measure show the performance for a specific decision threshold.
Therefore, when different classifiers are compared in terms of local metrics, the choice of the decision threshold becomes important. The decision threshold may be set to a fixed optimal value without considering the operating condition or based on operating conditions: the cost proportions or skew levels [28]. The performance metrics that can be maximized to set the decision threshold are accuracy, Brier score, AUC, expected cost and F-measure [28, 42]. To this aim, the classifiers are tested on a set of data called validation datasets that are independent from training and testing data.
As explained in Section 2.4, AUPR and F-measure are more suitable metrics to compare the performance of the classification systems when data is imbalanced. Therefore, in the experiments in this paper, AUPR is used to compare the performance of Boosting ensembles globally and F-measure with is used to compare the classifiers for a specific operating condition. AUPR shows the average value of precision for different values of recall (or TPR), and F2-measure shows the harmonic mean of precision and recall when a higher importance is given to recall.
The value of AUPR and F2-measure is averaged over 10 replications obtained by 2 5-fold cross validation. In our experiments, the decision threshold to obtain the F2-measure is set to the value that maximizes the value of F2-measure on the validation data for comparing the performance of different classification algorithms. An example is shown in Figure 7, the PR curve of an experiment on abalone9-18 dataset of KEEL collection on the validation data. In Figure 8, the ensembles are tested on a different test set and shows the value of F-measure when the optimal threshold is selected using the validation step described. is the value of F-measure when the combination function in Boosting ensembles is majority voting and the decisions of base classifiers are combined. It is observed that and may differ significantly and in most cases .
In our experiments, the performance of the proposed PBoost ensemble is also compared to state of the art Boosting ensembles in terms of computational complexity. Time complexity for SVM training depends on several factors including the number of training samples, the learning (optimization) algorithm and the number of features. The computational complexity of SVM implemented in LibSVM is evaluated in [38], as per iteration , where is the training set size, and is the number of features. The authors state that “the number of iterations may be higher than linear to the number of training data”. Therefore, the complexity is for some . This means that, time complexity for SVM training is not proportional to, but increases more than linearly with respect to the training set size.
In the proposed PRUS and baseline Boosting ensembles, is unknown and is identical in all algorithms. Each iteration of Boosting ensembles includes a validation step that should be added to training complexity to obtain the overall time complexity of learning process. Time complexity of the validation step , depends on the number of validation samples and the number of support vectors obtained from training each SVM. The reason is that, when an RBF SVM with support vectors is tested on a probe sample x, the value of is accumulated for all support vectors () and the sign of the resulting quantity determines the decision.
Table 3 shows the number of samples to train and validate these Boosting ensembles of the size . The number of validation samples in baseline Boosting ensembles is the same and equal to the overall number of training samples. However, the overall number of samples used for validation in PRUSBoost is calculated as:
[TABLE]
This value is less than that is the total number of validation samples in the sate of the art Boosting ensembles. Table 3 shows that the total number of training and validation samples in PRUS ensemble is the smallest one.
5 Results and Discussion
The performance of the proposed and state of the art ensemble learning methods are analysed for synthetic and video data in 4 parts: (1) accuracy and robustness over different levels of overlap and imbalance between design and test data and of using the proposed loss factor; (2) the performance of RUSBoost with and without progressive partitioning; (3) the combined impact of progressive partitioning and proposed loss factor; (4) the computation complexity during design and testing.
5.1 Results of Experiments with Synthetic Data
5.1.1 Impact of proposed loss factor
The performance of the baseline Boosting ensembles: AdaBoost , SMOTEBoost, RUSBoost, and RB-Boost are compared in Table 4 for different settings. In addition, Fβ is used to optimize loss factor calculation in these ensembles.
Given a fixed skew level of test data, the performance of all Boosting ensembles declines in terms of F-measure and AUPR as the overlap between positive and negative classes grows. This decline of performance is more significant when test data is imbalanced compared to the case where test data is balanced. In our experiments, changes in skew level of test data result in different number of misclassified negative samples and no change in the number of correctly classified positive samples. Therefore, even for the same level of overlap, the performance of all ensembles degrades, in terms of both F-measure and AUPR when testing on a more imbalanced data.
For the same level of overlap and different imbalance of training data ( and ) the performance of all Boosting ensembles is lower when imbalance of training data is lower. The reason is that less information is provided for training and also the skew level of training and test data has a greater difference. Overall, Table 4 shows that RUS is the most robust to changes in overlap and imbalance. Based on the Table 4, the following results are obtained. Using the proposed loss factor improves the performance of Ada with and for all , and with . However, the performance drops slightly with and and . Performance of SMT improves with for all , and it declines with and and as well as with and . Performance of RUS improves with and for all , and with and . However, performance of RUSBoost degrades slightly with and and . Performance of RB improves with for all , and with , but declines with . Based on the Table 4, the value of AUPR of Boosting ensembles after using the proposed loss factor does not change in most cases.
Using the proposed loss factor may improve the performance of the Boosting ensembles that rely on under-sampling of data in terms of F-measure, especially for more difficult problems with overlapping data. The performance of Boosting ensembles that involve up-sampling of positive samples does not improve significantly. However, the use of proposed loss factor has no impact on the global performance of these Boosting ensembles in terms of AUPR. Therefore, the use of proposed loss factor performs similarly to adjusting the decision threshold of the Boosting algorithms to better account for imbalance.
In Table 5, the performance of baseline ensembles and their variants for different values of is compared for . The goal is to evaluate the effect of the value of on improving the performance when the proposed loss factor is used. This Table shows the performance only when because the performance of baseline systems usually decline for higher skew levels of test data.
Evaluation is done in terms of the same Fβ-measure that is used in loss factor calculation. The results are shown in terms of both FD and Fop. FD is the value of F-measure when the decisions of base classifiers are combined in Boosting ensembles and Fop is the value of F-measure when the scores of base classifiers are combined in Boosting ensembles and the optimal decision threshold of each ensemble is set to the point that maximizes F-measure when that ensemble is validated on an independent set of data (see section 4.2.1.).
The performance of Ada improves for most value of in terms of both FD and Fop. Some improvements are seen for SMT and RB in terms of Fop, but FD tend to stay the same in most cases and decrease in some cases. The performance of RUS improves for and in terms of both FD and Fop, and the improvement tends to decrease for higher values. This was expected, since using higher values of to calculate F-measure means giving more importance to recall than precision. Therefore, the impact of imbalance is masked when higher values of is used and the performance may not change when the loss factor is calculated based on F-measure. For each of the classification systems, the same reason result in higher values of FD and Fop with higher values of . Comparing FD and Fop of each ensemble for each value of shows that selecting the proper decision threshold can improve the performance in terms of accuracy and robustness, especially for lower values of .
The results are not shown in terms of AUPR because AUPR does not change with variations in the value of , since the importance of recall and precision stays the same and equal in obtaining AUPR.
5.1.2 Impact of progressive partitioning in RUSBoost
In this section, progressive partitioning is integrated into RUS without the use of F-measure in loss factor calculation. It is observed in Table 6 that robustness of RUS improves significantly after using this method of sampling. Indeed, using all samples for training through partitioning avoids loss of information and may improve the classification accuracy. In addition, validating on different imbalance levels of data increases the robustness to variations in the imbalance level of test data.
The performance of PRUS and PCUSi is significantly better than RUS in terms of both F-measure and AUPR, especially with and . Partitioning these datasets using PCUSi, and random under-sampling without replacement is more effective in improving the performance of RUS.
5.1.3 Impact of progressive partitioning and loss factor combined
In this section progressive partitioning and the proposed loss factor are integrated into RUS algorithm, resulting in PRUS-F, PCUS-F, and PCUSi-F. Over all ranges of skew and overlap in Table 7, PCUSi-F outperforms other classification systems and PRUS-F takes the second place, especially for higher levels of imbalance in test data. Combining the use of F-measure and progressive partitioning is more effective in increasing performance and robustness compared to using each of them independently because accuracy and robustness to imbalance improve at the same time, not separately, during learning process. If the negative class is partitioned a priori (CUSi), PBoost performs significantly better than the case when general partitioning techniques (RUS and CUS) are used.
5.2 Results of Experiments with Video Data
Similarly to the synthetic data sets, the results of experiments on video dataset are shown in three parts, assessing the impact of: (1) using the proposed loss factor on the performance of baseline Boosting ensembles, (2) integrating progressive partitioning into RUS, and (3) using the proposed loss factor and progressive partitioning compared with the baseline and state of the art Boosting ensembles.
From Table 8, the performance level of all ensembles is lower when the skew level of training data is higher. This is despite the fact that when the imbalance of training data is lower, the data that is used to test classifiers contain samples from some individuals that are not in the training data. Using the proposed loss factor improves the performance of Ada, RUS and SMT in terms of F-measure, and has no impact on the performance of RB in most cases of skew between classes in training and testing data. The performance of these ensembles after using the proposed loss factor does not change in terms of AUPR and therefore they are not shown here. In fact, the use of proposed loss factor performs similarly to adjusting the decision threshold of the Boosting algorithms to better account for imbalance and therefore may improve the performance only in terms of local performance metrics like F-measure.
After integrating the progressive partitioning in RUS using PRUS and PTUS, the performance of RUS increases and becomes more robust in terms of both F-measure and AUPR (see Table 9), especially when TUS is used for partitioning because validating base classifiers on different imbalance levels of imbalance result in more robust classification systems and using all samples for training through partitioning avoids loss of information and may improve the classification accuracy.
Comparing the performance of final PBoost variants with baseline ensembles in Table 10, PTUS-F outperforms all other approaches in terms of F-measure and AUPR. In some cases, RB performs the same as PTUS-F. From these results, it is observed that combining the use of F-measure and integration of progressive partitioning, is more effective in increasing performance and robustness compared to using each of them independently. In the experiments on the video data, trajectory under-sampling is more effective when used in PBoost compared to random under-sampling without replacement and cluster under-sampling. This is the case when partitions of negative class are known a priori.
5.3 Results of Experiments with KEEL Collection
In Table 11, the performance of baseline and proposed Boosting ensembles is compared in terms of F2-measure and AUPR for experiments with 21 KEEL datasets. The second column of the table shows the imbalance level of training and test data in each dataset that ranges between 1:9 and 1:29. In this table, for each dataset, the best values are bold and second-best values are italic-bold to show the first and second best classifiers, respectively. In terms of F-measure, PCUS has one of the two highest values for 17 datasets, RB has one of the two highest values for 15 datasets and PRUS has one of the two highest values for 13 datasets. RB is the best classifier for 14 datasets while PCUS and PRUS are the best for 5 and 7 datasets, respectively. In terms of AUPR, PCUS and RB have one of the two highest values for 17 datasets, and PRUS has one of the two highest values for 8 datasets. RB is the best classifier for 15 datasets while PCUS and PRUS are the best for 5 datasets. In the cases when no natural data partitioning is known a priori, clustering with k-means is more effective than random under-sampling because base classifiers are trained on different parts of feature space and therefore the ensemble have more diversity compared to the case when the base classifiers are trained on samples from all parts of feature space. More sophisticated clustering methods like kernel k-means and spectral clustering may be more suitable.
5.4 Computational Complexity
In this section the time complexity needed to design and test the proposed and baseline Boosting ensembles are compared. To compare the training time and memory cost of these ensembles, the number of training samples is counted and to compare their validation time and memory cost the number of validation samples and the number of support vectors of base classifiers are considered.
Figure 9 show the results obtained with data set in our experiments. The number of training and validation samples, the average number of support vectors, and overall number of evaluations of the kernel function () is presented in Figure 9(a)-(d) to estimate and compare design time of the proposed and baseline Boosting ensembles. To compare the complexity of these classification systems during testing with a probe sample x, we compared the overall number of support vectors in these ensembles in Figure 9(e) because computing each SVM output requires evaluations of the kernel function.
Given as the number of samples to train the classifier in the ensemble, Figure 9(a) shows . In Figure 9(b), is presented, where is the number of samples that the classifier in the ensemble is validated with. Average number of support vectors in classifiers of the ensembles are shown in Figure 9(c). Given as the number of support vectors obtained after training the classifier in the ensemble, Figure 9(d) shows for each ensemble.
In terms of training (see Figure 9(a)), PRUS and RUS are under-sampling ensembles and have the lowest computational cost, while SMT and RB-Boost include up-sampling and are significantly more costly. Total number of validation samples is equal for Ada, SMT, RUS and RB, and total number of validation samples is less with PRUS (see Figure 9(b)). The average number of support vectors is higher for SMT (see Figure 9(c)) because the base classifiers in this ensemble are trained on higher number of samples. Therefore, SMT is the most costly method, in terms of validation (see Figure 9(d)). Note that time and memory required for partitioning in PRUS, and generating synthetic samples in SMT and RB-Boost is neglected here. Nevertheless, PRUS is the most efficient ensemble technique in terms of designing memory and time complexity.
The number of training and validation samples as well as the average number of support vectors is smaller with PRUS and therefore, PRUS is less costly in terms of design time and memory complexity.
In terms of testing time complexity (see Figure 9(e)) PRUS and RUS have the lowest number of evaluations of the kernel function per probe sample. RB-Boost and SMT have the highest number of evaluations of the kernel function per probe sample.
Although Ada is given the same ensemble size, it fails to generate enough classifiers 333AdaBoost failed many times to be generated because the training subset in step 2.i of Algo. 1 may contain only negative class samples during sampling or weight update in step 2.vii may lead to that due to unsuitable loss factor calculation in step 2.ivv. Nevertheless, only successful attempts are considered. and consequently result in smaller number of support vectors. Therefore, the total number of validation and testing processes of Ada is lower than expected.
5.5 Summary of Results
As a summary of results on synthetic, video and KEEL datasets, we observed that:
Using the proposed loss factor calculation may reduce the bias of performance in Boosting ensembles and increase the accuracy. 2. 2.
Partitioning improves the performance of RUS in all cases in terms of both accuracy and robustness to imbalance. 3. 3.
Integrating both partitioning and the proposed loss factor outperforms state of the art Boosting ensembles, relying on the choice of partitioning technique for each dataset such that:
- (a)
With synthetic data, PCUSi-F outperforms all systems in terms of both F-measure and AUPR, while PRUS and PCUS outperform RUS in most cases of skew and overlap between classes. 2. (b)
With the video data, PTUS is more accurate than the state of the art ensembles, and PRUS as well as PCUS. 3. (c)
PCUS is one of the best two classifiers in most KEEL problems, and performs very closely to RB in terms of both F-measure and AUPR. 4. 4.
PBoost is computationally less costly than the state of the art Boosting ensembles in terms of computational complexity.
Therefore, PBoost is an effective approach in correct classification of data when data is imbalanced in comparison to the state of the art Boosting ensembles. This method relies on the choice of partitioning technique for each dataset and performs significantly better when a more suitable partitioning technique is used. In problems that the natural clusters are known, the performance is better than using the general partitioning methods such as random under-sampling without replacement or k-means clustering. Therefore, PBoost can be more efficient than baseline Boosting ensembles considering both accuracy and complexity factors.
6 Conclusion
In this paper, a new Boosting ensemble algorithm named as PBoost is proposed to address imbalance based on the idea of modifying RUSBoost by (1) under-sampling the majority class using partitional techniques, (2) validating classifiers on a growing validation subset, and (3) using a more suitable loss factor calculation. The partitions enter the Boosting process progressively for designing classifiers over iterations to avoid information loss and to maintain diversity among them. Validating base classifiers on a growing number of negative samples makes the PBoost ensembles more robust to possible skew levels of data during operations in addition to lowering the computational complexity. The new loss factor defined in this Boosting ensemble handles bias of performance towards negative class, and guides the Boosting process in a more effective direction with the purpose of correctly classifying both classes. Experiments show that PBoost may perform differently with different techniques of partitioning for each dataset such that more suitable clustering result in better performance. Nevertheless, regardless of the partitioning method, the PBoost ensembles perform comparably to state of the art Boosting ensembles in terms of accuracy. In addition, PBoost has significantly lower computational complexity in both designing and testing stages compared to state of the art Boosting ensembles. The applicability of PBoost to multi-class classification problems and deployment of more sophisticated clustering methods in PBoost can be further investigated in a future work.
Acknowledgement
This work was partially supported by the Natural Sciences and Engineering Research Council of Canada and Mitacs.
References
- [1]
M. Galar, A. Fernández, E. Barrenechea, H. Bustince, F. Herrera, An overview of ensemble methods for binary classifiers in multi-class problems: Experimental study on one-vs-one and one-vs-all schemes, Pattern Recognition 44 (8) (2011) 1761–1776.
- [2]
S. Wang, X. Yao, Multiclass imbalance problems: Analysis and potential solutions, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 42 (4) (2012) 1119–1130.
- [3]
H. He, E. A. Garcia, Learning from imbalanced data, Knowledge and Data Engineering, IEEE Transactions on 21 (9) (2009) 1263–1284.
- [4]
M. Galar, A. Fernandez, E. Barrenechea, H. Bustince, F. Herrera, A review on ensembles for the class imbalance problem: bagging-, boosting-, and hybrid-based approaches, Systems, Man, and Cybernetics, Part C: Applications and Reviews, IEEE Transactions on 42 (4) (2012) 463–484.
- [5]
P. V. Radtke, E. Granger, R. Sabourin, D. O. Gorodnichy, Skew-sensitive boolean combination for adaptive ensembles–an application to face recognition in video surveillance, Information Fusion 20 (2014) 31–48.
- [6]
M. De-la Torre, E. Granger, R. Sabourin, D. O. Gorodnichy, Adaptive skew-sensitive ensembles for face recognition in video surveillance, Pattern Recognition 48 (11) (2015) 3385–3406.
- [7]
R. Soleymani, E. Granger, G. Fumera, Classifier ensembles with trajectory under-sampling for face re-identification, in: Proceedings of the International Conference on Pattern Recognition Applications and Methods-Volume 1, SCITEPRESS-Science and Technology Publications, Lda, 2016, pp. 97–108.
- [8]
Y. Freund, R. E. Schapire, A desicion-theoretic generalization of on-line learning and an application to boosting, in: Computational learning theory, Springer, 1995, pp. 23–37.
- [9]
Y. Freund, R. E. Schapire, et al., Experiments with a new boosting algorithm, in: Machine Learning and Applications, 1996. ICMLA’96 International Conference on, 1996.
- [10]
N. V. Chawla, A. Lazarevic, L. O. Hall, K. W. Bowyer, Smoteboost: Improving prediction of the minority class in boosting, in: European Conference on Principles of Data Mining and Knowledge Discovery, Springer, 2003, pp. 107–119.
- [11]
S. Hu, Y. Liang, L. Ma, Y. He, Msmote: improving classification performance when training data is imbalanced, in: Proceedings of the 2009 Second International Workshop on Computer Science and Engineering, Vol. 2, Citeseer, 2009, pp. 13–17.
- [12]
D. Mease, A. Wyner, A. Buja, Cost-weighted boosting with jittering and over/under-sampling: Jous-boost, J. Machine Learning Research 8 (2007) 409–439.
- [13]
H. Guo, H. L. Viktor, Learning from imbalanced data sets with boosting and data generation: the databoost-im approach, ACM SIGKDD Explorations Newsletter 6 (1) (2004) 30–39.
- [14]
C. Seiffert, T. M. Khoshgoftaar, J. Van Hulse, A. Napolitano, Rusboost: A hybrid approach to alleviating class imbalance, Systems, Man and Cybernetics, Part A: Systems and Humans, IEEE Transactions on 40 (1) (2010) 185–197.
- [15]
M. Galar, A. Fernández, E. Barrenechea, F. Herrera, Eusboost: Enhancing ensembles for highly imbalanced data-sets by evolutionary undersampling, Pattern Recognition 46 (12) (2013) 3460–3471.
- [16]
J. F. Díez-Pastor, J. J. Rodríguez, C. García-Osorio, L. I. Kuncheva, Random balance: Ensembles of variable priors classifiers for imbalanced data, Knowledge-Based Systems 85 (2015) 96–111.
- [17]
W. Fan, S. J. Stolfo, J. Zhang, P. K. Chan, Adacost: misclassification cost-sensitive boosting, in: Machine Learning and Applications, 1999. ICMLA’99 International Conference on, 1999.
- [18]
K. M. Ting, A comparative study of cost-sensitive boosting algorithms, in: Machine Learning and Applications, 2000. International Conference on, Citeseer, 2000.
- [19]
Y. Sun, M. S. Kamel, A. K. Wong, Y. Wang, Cost-sensitive boosting for classification of imbalanced data, Pattern Recognition 40 (12) (2007) 3358–3378.
- [20]
M. V. Joshi, V. Kumar, R. C. Agarwal, Evaluating boosting algorithms to classify rare classes: Comparison and improvements, in: Data Mining, 2001. ICDM 2001, Proceedings IEEE International Conference on, IEEE, 2001, pp. 257–264.
- [21]
M.-J. Kim, D.-K. Kang, H. B. Kim, Geometric mean based boosting algorithm with over-sampling to resolve data imbalance problem for bankruptcy prediction, Expert Systems with Applications 42 (3) (2015) 1074–1082.
- [22]
L. Rokach, Ensemble-based classifiers, Artificial Intelligence Review 33 (1-2) (2010) 1–39.
- [23]
S. García, F. Herrera, Evolutionary undersampling for classification with imbalanced datasets: Proposals and taxonomy, Evolutionary computation 17 (3) (2009) 275–306.
- [24]
R. Yan, Y. Liu, R. Jin, A. Hauptmann, On predicting rare classes with svm ensembles in scene classification, in: Acoustics, Speech, and Signal Processing, 2003. Proceedings.(ICASSP’03). 2003 IEEE International Conference on, Vol. 3, IEEE, 2003, pp. III–21.
- [25]
Q. Li, B. Yang, Y. Li, N. Deng, L. Jing, Constructing support vector machine ensemble with segmentation for imbalanced datasets, Neural Computing and Applications 22 (1) (2013) 249–256.
- [26]
T. Fawcett, An introduction to roc analysis, Pattern recognition letters 27 (8) (2006) 861–874.
- [27]
V. Garcıa, R. Mollineda, J. Sánchez, Theoretical analysis of a performance measure for imbalanced data, in: Proceedings of the 20th International Conference on Pattern Recognition (ICPR’10), 2010, pp. 617–620.
- [28]
J. Hernández-Orallo, P. Flach, C. Ferri, A unified view of performance metrics: translating threshold choice into expected classification loss, Journal of Machine Learning Research 13 (Oct) (2012) 2813–2869.
- [29]
P. Flach, M. Kull, Precision-recall-gain curves: Pr analysis done right, in: Advances in Neural Information Processing Systems, 2015, pp. 838–846.
- [30]
R. Xu, D. Wunsch, Survey of clustering algorithms, IEEE Transactions on neural networks 16 (3) (2005) 645–678.
- [31]
R. Goh, L. Liu, X. Liu, T. Chen, The cmu face in action (fia) database, in: Analysis and Modelling of Faces and Gestures, 2005, pp. 255–263.
- [32]
J. Alcalá, A. Fernández, J. Luengo, J. Derrac, S. García, L. Sánchez, F. Herrera, Keel data-mining software tool: Data set repository, integration of algorithms and experimental analysis framework, Journal of Multiple-Valued Logic and Soft Computing 17 (2-3) (2010) 255–287.
- [33]
M. De-la Torre, E. Granger, P. V. Radtke, R. Sabourin, D. O. Gorodnichy, Partially-supervised learning from facial trajectories for face recognition in video surveillance, Information Fusion 24 (2015) 31–53.
- [34]
C. Pagano, E. Granger, R. Sabourin, G. L. Marcialis, F. Roli, Adaptive ensembles for face recognition in changing video surveillance environments, Information Sciences 286 (2014) 75–101.
- [35]
J. W. Sammon, A nonlinear mapping for data structure analysis, IEEE Transactions on computers 100 (5) (1969) 401–409.
- [36]
P. Viola, M. Jones, Rapid object detection using a boosted cascade of simple features, in: Computer Vision and Pattern Recognition, 2001. CVPR 2001. Proceedings of the 2001 IEEE Computer Society Conference on, Vol. 1, IEEE, 2001, pp. I–511.
- [37]
T. Ojala, M. Pietikainen, T. Maenpaa, Multiresolution gray-scale and rotation invariant texture classification with local binary patterns, Pattern Analysis and Machine Intelligence, IEEE Transactions on 24 (7) (2002) 971–987.
- [38]
C.-C. Chang, C.-J. Lin, Libsvm: a library for support vector machines, ACM Transactions on Intelligent Systems and Technology (TIST) 2 (3) (2011) 27.
- [39]
X. Li, L. Wang, E. Sung, Adaboost with svm-based component classifiers, Engineering Applications of Artificial Intelligence 21 (5) (2008) 785–795.
- [40]
J. C. Dunn, A fuzzy relative of the isodata process and its use in detecting compact well-separated clusters.
- [41]
V. López, A. Fernández, S. García, V. Palade, F. Herrera, An insight into classification with imbalanced data: Empirical results and current trends on using data intrinsic characteristics, Information Sciences 250 (2013) 113–141.
- [42]
Z. C. Lipton, C. Elkan, B. Naryanaswamy, Optimal thresholding of classifiers to maximize f1 measure, in: Joint European Conference on Machine Learning and Knowledge Discovery in Databases, Springer, 2014, pp. 225–239.
The reference list from the paper itself. Each links out to its DOI / PubMed record.
- 1[1] M. Galar, A. Fernández, E. Barrenechea, H. Bustince, F. Herrera, An overview of ensemble methods for binary classifiers in multi-class problems: Experimental study on one-vs-one and one-vs-all schemes, Pattern Recognition 44 (8) (2011) 1761–1776.
- 2[2] S. Wang, X. Yao, Multiclass imbalance problems: Analysis and potential solutions, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 42 (4) (2012) 1119–1130.
- 3[3] H. He, E. A. Garcia, Learning from imbalanced data, Knowledge and Data Engineering, IEEE Transactions on 21 (9) (2009) 1263–1284.
- 4[4] M. Galar, A. Fernandez, E. Barrenechea, H. Bustince, F. Herrera, A review on ensembles for the class imbalance problem: bagging-, boosting-, and hybrid-based approaches, Systems, Man, and Cybernetics, Part C: Applications and Reviews, IEEE Transactions on 42 (4) (2012) 463–484.
- 5[5] P. V. Radtke, E. Granger, R. Sabourin, D. O. Gorodnichy, Skew-sensitive boolean combination for adaptive ensembles–an application to face recognition in video surveillance, Information Fusion 20 (2014) 31–48.
- 6[6] M. De-la Torre, E. Granger, R. Sabourin, D. O. Gorodnichy, Adaptive skew-sensitive ensembles for face recognition in video surveillance, Pattern Recognition 48 (11) (2015) 3385–3406.
- 7[7] R. Soleymani, E. Granger, G. Fumera, Classifier ensembles with trajectory under-sampling for face re-identification, in: Proceedings of the International Conference on Pattern Recognition Applications and Methods-Volume 1, SCITEPRESS-Science and Technology Publications, Lda, 2016, pp. 97–108.
- 8[8] Y. Freund, R. E. Schapire, A desicion-theoretic generalization of on-line learning and an application to boosting, in: Computational learning theory, Springer, 1995, pp. 23–37.