Predictive Volatility of Machine Learning in Micro-Samples: A Regularised Assessment of Regional Poverty

TL;DR

This study compares various statistical and machine learning models to identify the most reliable methods for analyzing poverty drivers in small, collinear regional datasets, emphasizing the effectiveness of regularised linear models.

Contribution

It demonstrates that regularised linear models outperform complex machine learning ensembles in small, collinear regional datasets for poverty analysis.

Findings

Simple linear shrinkage models outperform complex ensembles in predictive accuracy.

ICT skills are consistently identified as a key factor in reducing poverty.

Overfitting is prevalent in complex machine learning models like BART on small regional data.

Abstract

Identifying the structural drivers of poverty in regional datasets is frequently hindered by small sample sizes and high multidimensional collinearity, which can result in unstable and misleading policy advice. This paper evaluates the provincial causes of poverty in Indonesia by addressing these specific statistical hazards. We employ a rigorous model-comparison framework designed for small samples ($n=34$) with high collinearity, comparing standard linear models with frequentist penalisation, Bayesian shrinkage priors, an adjusted spatial intrinsic conditionally autoregressive (ICAR) model, and complex machine learning ensembles. To ensure a robust evaluation, we measure predictive performance using strict Leave-One-Out Cross-Validation (LOOCV). The results demonstrate that algorithmic complexity is inherently risky in regional datasets: simple linear shrinkage models (Ridge, Elastic Net, LASSO) achieve the superior out-of-sample prediction, whereas complex ensembles like BART suffer from severe overfitting. Across all successful regularised models, ICT skills consistently emerge as the most stable proxy for lower provincial poverty. The primary contribution of this paper is demonstrating that, in data-constrained regional analysis, parametrically regularised linear shrinkage provides a more reliable mathematical foundation for isolating structural development priorities, such as ICT, than either naive OLS or unconstrained machine learning.