On the combined effect of class imbalance and concept complexity in deep learning

TL;DR

This study systematically evaluates how deep learning models handle class imbalance, concept complexity, and data scarcity, revealing that deeper architectures offer limited advantages in challenging scenarios, especially with real-world data.

Contribution

It provides a comprehensive analysis of deep learning performance under classical challenge conditions, highlighting limitations of depth in addressing class overlap and data scarcity.

Findings

Deeper architectures help with structural concept complexity.

Deeper models do not significantly improve handling of class overlap.

In real-world datasets, deeper models often overfit and perform worse.

Abstract

Structural concept complexity, class overlap, and data scarcity are some of the most important factors influencing the performance of classifiers under class imbalance conditions. When these effects were uncovered in the early 2000s, understandably, the classifiers on which they were demonstrated belonged to the classical rather than Deep Learning categories of approaches. As Deep Learning is gaining ground over classical machine learning and is beginning to be used in critical applied settings, it is important to assess systematically how well they respond to the kind of challenges their classical counterparts have struggled with in the past two decades. The purpose of this paper is to study the behavior of deep learning systems in settings that have previously been deemed challenging to classical machine learning systems to find out whether the depth of the systems is an asset in such settings. The results in both artificial and real-world image datasets (MNIST Fashion, CIFAR-10) show that these settings remain mostly challenging for Deep Learning systems and that deeper architectures seem to help with structural concept complexity but not with overlap challenges in simple artificial domains. Data scarcity is not overcome by deeper layers, either. In the real-world image domains, where overfitting is a greater concern than in the artificial domains, the advantage of deeper architectures is less obvious: while it is observed in certain cases, it is quickly cancelled as models get deeper and perform worse than their shallower counterparts.