Regularized Meta-Learning for Improved Generalization

TL;DR

This paper introduces a regularized meta-learning framework that enhances ensemble model generalization by reducing redundancy, stabilizing weighting, and improving computational efficiency through a multi-stage pipeline.

Contribution

It proposes a novel redundancy-aware meta-learning pipeline with de-duplication, meta-feature augmentation, and regularized meta-models, outperforming traditional stacking methods.

Findings

Achieves lower RMSE (8.582) on the benchmark compared to simple averaging and Ridge stacking.

Reduces the effective condition number of the meta-design matrix by 53.7%.

Demonstrates consistent improvements from de-duplication, meta-features, and blending in ablation studies.

Abstract

Deep ensemble methods often improve predictive performance, yet they suffer from three practical limitations: redundancy among base models that inflates computational cost and degrades conditioning, unstable weighting under multicollinearity, and overfitting in meta-learning pipelines. We propose a regularized meta-learning framework that addresses these challenges through a four-stage pipeline combining redundancy-aware projection, statistical meta-feature augmentation, and cross-validated regularized meta-models (Ridge, Lasso, and ElasticNet). Our multi-metric de-duplication strategy removes near-collinear predictors using correlation and MSE thresholds ($\tau_{\text{corr}}=0.95$), reducing the effective condition number of the meta-design matrix while preserving predictive diversity. Engineered ensemble statistics and interaction terms recover higher-order structure unavailable to raw prediction columns. A final inverse-RMSE blending stage mitigates regularizer-selection variance. On the Playground Series S6E1 benchmark (100K samples, 72 base models), the proposed framework achieves an out-of-fold RMSE of 8.582, improving over simple averaging (8.894) and conventional Ridge stacking (8.627), while matching greedy hill climbing (8.603) with substantially lower runtime (4 times faster). Conditioning analysis shows a 53.7\% reduction in effective matrix condition number after redundancy projection. Comprehensive ablations demonstrate consistent contributions from de-duplication, statistical meta-features, and meta-ensemble blending. These results position regularized meta-learning as a stable and deployment-efficient stacking strategy for high-dimensional ensemble systems.