Estimation of Over-parameterized Models from an Auto-Modeling Perspective

TL;DR

This paper introduces a novel paradigm for fitting over-parameterized models by focusing on future observations, employing an adaptive duality function and imputation method, with applications to various statistical and machine learning problems.

Contribution

It proposes a new estimation framework based on future observation fitting, including an adaptive imputation technique, and demonstrates its effectiveness across multiple applications.

Findings

Superior performance in diverse applications

Effective imputation method using adaptive bootstrap

In-depth theoretical analysis of the approach

Abstract

From a model-building perspective, we propose a paradigm shift for fitting over-parameterized models. Philosophically, the mindset is to fit models to future observations rather than to the observed sample. Technically, given an imputation method to generate future observations, we fit over-parameterized models to these future observations by optimizing an approximation of the desired expected loss function based on its sample counterpart and an adaptive $\textit{duality function}$. The required imputation method is also developed using the same estimation technique with an adaptive $m$-out-of-$n$ bootstrap approach. We illustrate its applications with the many-normal-means problem, $n < p$ linear regression, and neural network-based image classification of MNIST digits. The numerical results demonstrate its superior performance across these diverse applications. While primarily expository, the paper conducts an in-depth investigation into the theoretical aspects of the topic. It concludes with remarks on some open problems.