Minimal Variance Model Aggregation: A principled, non-intrusive, and versatile integration of black box models

TL;DR

The paper introduces MEVA, a flexible, data-driven framework for aggregating black box models to improve accuracy and robustness, applicable across diverse domains like data science and PDEs.

Contribution

It presents a novel, non-intrusive aggregation method that optimizes variance reduction, outperforming error minimization approaches in robustness and versatility.

Findings

MEVA improves accuracy across multiple applications.

MVA provides more robust estimates than MEA.

Framework is compatible with various model types and domains.

Abstract

Whether deterministic or stochastic, models can be viewed as functions designed to approximate a specific quantity of interest. We introduce Minimal Empirical Variance Aggregation (MEVA), a data-driven framework that integrates predictions from various models, enhancing overall accuracy by leveraging the individual strengths of each. This non-intrusive, model-agnostic approach treats the contributing models as black boxes and accommodates outputs from diverse methodologies, including machine learning algorithms and traditional numerical solvers. We advocate for a point-wise linear aggregation process and consider two methods for optimizing this aggregate: Minimal Error Aggregation (MEA), which minimizes the prediction error, and Minimal Variance Aggregation (MVA), which focuses on reducing variance. We prove a theorem showing that MVA can be more robustly estimated from data than MEA, making MEVA superior to Minimal Empirical Error Aggregation (MEEA). Unlike MEEA, which interpolates target values directly, MEVA formulates aggregation as an error estimation problem, which can be performed using any backbone learning paradigm. We demonstrate the versatility and effectiveness of our framework across various applications, including data science and partial differential equations, illustrating its ability to significantly enhance both robustness and accuracy.