Improving Performance in Classification Tasks with LCEN and the Weighted Focal Differentiable MCC Loss

TL;DR

This paper introduces a modified LCEN algorithm for classification that maintains interpretability, achieves high performance, and effectively reduces features, while also evaluating a new diffMCC loss function that outperforms traditional loss functions.

Contribution

The work adapts LCEN for classification tasks and introduces the weighted focal differentiable MCC loss, demonstrating improved performance and feature sparsity.

Findings

LCEN achieves high macro F1 and MCC scores across datasets.

LCEN models eliminate an average of 56% of input features.

Models trained with diffMCC loss outperform those with cross-entropy loss.

Abstract

The LASSO-Clip-EN (LCEN) algorithm was previously introduced for nonlinear, interpretable feature selection and machine learning. However, its design and use was limited to regression tasks. In this work, we create a modified version of the LCEN algorithm that is suitable for classification tasks and maintains its desirable properties, such as interpretability. This modified LCEN algorithm is evaluated on four widely used binary and multiclass classification datasets. In these experiments, LCEN is compared against 10 other model types and consistently reaches high test-set macro F$_1$ score and Matthews correlation coefficient (MCC) metrics, higher than that of the majority of investigated models. LCEN models for classification remain sparse, eliminating an average of 56% of all input features in the experiments performed. Furthermore, LCEN-selected features are used to retrain all models using the same data, leading to statistically significant performance improvements in three of the experiments and insignificant differences in the fourth when compared to using all features or other feature selection methods. Simultaneously, the weighted focal differentiable MCC (diffMCC) loss function is evaluated on the same datasets. Models trained with the diffMCC loss function are always the best-performing methods in these experiments, and reach test-set macro F$_1$ scores that are, on average, 4.9% higher and MCCs that are 8.5% higher than those obtained by models trained with the weighted cross-entropy loss. These results highlight the performance of LCEN as a feature selection and machine learning algorithm also for classification tasks, and how the diffMCC loss function can train very accurate models, surpassing the weighted cross-entropy loss in the tasks investigated.