A naive aggregation algorithm for improving generalization in a class of learning problems

TL;DR

This paper introduces a simple aggregation algorithm for expert advice in high-dimensional nonlinear function estimation, aiming to improve model generalization by combining expert estimates through sequential decision strategies.

Contribution

It proposes a naive aggregation method with theoretical conditions for optimal consensus estimation in expert advice learning problems.

Findings

Algorithm improves generalization over individual experts

Conditions for effective aggregation are established

Numerical results demonstrate improved nonlinear regression performance

Abstract

In this brief paper, we present a naive aggregation algorithm for a typical learning problem with expert advice setting, in which the task of improving generalization, i.e., model validation, is embedded in the learning process as a sequential decision-making problem. In particular, we consider a class of learning problem of point estimations for modeling high-dimensional nonlinear functions, where a group of experts update their parameter estimates using the discrete-time version of gradient systems, with small additive noise term, guided by the corresponding subsample datasets obtained from the original dataset. Here, our main objective is to provide conditions under which such an algorithm will sequentially determine a set of mixing distribution strategies used for aggregating the experts' estimates that ultimately leading to an optimal parameter estimate, i.e., as a consensus solution for all experts, which is better than any individual expert's estimate in terms of improved generalization or learning performances. Finally, as part of this work, we present some numerical results for a typical case of nonlinear regression problem.