On the utility of feature selection in building two-tier decision trees

TL;DR

This paper investigates how feature selection impacts two-tier decision trees, revealing that removing interfering features can significantly enhance performance, especially with limited domain knowledge, by up to 24 times.

Contribution

It demonstrates that eliminating interfering features can substantially improve two-tier decision tree performance, expanding feature selection methods for resource-rich scenarios.

Findings

Removing interfering features can improve performance by up to 24 times.

There is a negative correlation between initial performance and improvement after feature removal.

Performance gains are greater when less domain knowledge is learned.

Abstract

Nowadays, feature selection is frequently used in machine learning when there is a risk of performance degradation due to overfitting or when computational resources are limited. During the feature selection process, the subset of features that are most relevant and least redundant is chosen. In recent years, it has become clear that, in addition to relevance and redundancy, features' complementarity must be considered. Informally, if the features are weak predictors of the target variable separately and strong predictors when combined, then they are complementary. It is demonstrated in this paper that the synergistic effect of complementary features mutually amplifying each other in the construction of two-tier decision trees can be interfered with by another feature, resulting in a decrease in performance. It is demonstrated using cross-validation on both synthetic and real datasets, regression and classification, that removing or eliminating the interfering feature can improve performance by up to 24 times. It has also been discovered that the lesser the domain is learned, the greater the increase in performance. More formally, it is demonstrated that there is a statistically significant negative rank correlation between performance on the dataset prior to the elimination of the interfering feature and performance growth after the elimination of the interfering feature. It is concluded that this broadens the scope of feature selection methods for cases where data and computational resources are sufficient.