DUA-D2C: Dynamic Uncertainty Aware Method for Overfitting Remediation in Deep Learning

TL;DR

DUA-D2C is a novel method that dynamically weights ensemble models based on their uncertainty and validation performance, significantly improving overfitting mitigation and generalization in deep learning.

Contribution

It introduces a dynamic, uncertainty-aware aggregation technique for the Divide2Conquer framework, enhancing overfitting remediation in deep learning models.

Findings

Significant improvement in generalization across multiple benchmark datasets.

Effective combination with existing regularization methods.

Enhanced decision boundary robustness and reduced overfitting.

Abstract

Overfitting remains a significant challenge in deep learning, often arising from data outliers, noise, and limited training data. To address this, the Divide2Conquer (D2C) method was previously proposed, which partitions training data into multiple subsets and trains identical models independently on each. This strategy enables learning more consistent patterns while minimizing the influence of individual outliers and noise. However, D2C's standard aggregation typically treats all subset models equally or based on fixed heuristics (like data size), potentially underutilizing information about their varying generalization capabilities. Building upon this foundation, we introduce Dynamic Uncertainty-Aware Divide2Conquer (DUA-D2C), an advanced technique that refines the aggregation process. DUA-D2C dynamically weights the contributions of subset models based on their performance on a shared validation set, considering both accuracy and prediction uncertainty. This intelligent aggregation allows the central model to preferentially learn from subsets yielding more generalizable and confident edge models, thereby more effectively combating overfitting. Empirical evaluations on benchmark datasets spanning multiple domains demonstrate that DUA-D2C significantly improves generalization. Our analysis includes evaluations of decision boundaries, loss curves, and other performance metrics, highlighting the effectiveness of DUA-D2C. This study demonstrates that DUA-D2C improves generalization performance even when applied on top of other regularization methods, establishing it as a theoretically grounded and effective approach to combating overfitting in modern deep learning. Our codes are publicly available at: https://github.com/Saiful185/DUA-D2C.