How Ensemble Learning Balances Accuracy and Overfitting: A Bias-Variance Perspective on Tabular Data

TL;DR

This paper investigates how ensemble learning balances accuracy and overfitting on tabular data, revealing conditions under which ensembles improve performance and control variance, with practical insights for model selection.

Contribution

It provides a bias-variance perspective on ensemble methods for tabular data, identifying when ensembles enhance accuracy without overfitting and introducing dataset complexity indicators.

Findings

Ensembles reduce variance and improve accuracy on nonlinear, complex data.

Linear models perform well on simple, clean data with minimal ensemble benefit.

Regularization is necessary for ensembles on noisy or imbalanced datasets.

Abstract

Ensemble models often achieve higher accuracy than single learners, but their ability to maintain small generalization gaps is not always well understood. This study examines how ensembles balance accuracy and overfitting across four tabular classification tasks: Breast Cancer, Heart Disease, Pima Diabetes, and Credit Card Fraud. Using repeated stratified cross validation with statistical significance testing, we compare linear models, a single decision tree, and nine ensemble methods. The results show that ensembles can reach high accuracy without large gaps by reducing variance through averaging or controlled boosting. On nearly linear and clean data, linear models already generalize well and ensembles offer little additional benefit. On datasets with meaningful nonlinear structure, tree based ensembles increase test accuracy by 5 to 7 points while keeping gaps below 3 percent. On noisy or highly imbalanced datasets, ensembles remain competitive but require regularization to avoid fitting noise or majority class patterns. We also compute simple dataset complexity indicators, such as linearity score, Fisher ratio, and noise estimate, which explain when ensembles are likely to control variance effectively. Overall, the study provides a clear view of how and when ensembles maintain high accuracy while keeping overfitting low, offering practical guidance for model selection in real world tabular applications.