On Defining Neural Averaging

TL;DR

This paper introduces Amortized Model Ensembling (AME), a data-free framework for neural network averaging that generalizes model soup and improves out-of-distribution performance.

Contribution

It proposes a novel data-free meta-optimization approach for neural averaging, extending model soup with more expressive ensembling strategies.

Findings

AME outperforms individual models and model soup baselines

It enhances out-of-distribution generalization

Provides a principled approach to data-free neural weight aggregation

Abstract

What does it even mean to average neural networks? We investigate the problem of synthesizing a single neural network from a collection of pretrained models, each trained on disjoint data shards, using only their final weights and no access to training data. In forming a definition of neural averaging, we take insight from model soup, which appears to aggregate multiple models into a singular model while enhancing generalization performance. In this work, we reinterpret model souping as a special case of a broader framework: Amortized Model Ensembling (AME) for neural averaging, a data-free meta-optimization approach that treats model differences as pseudogradients to guide neural weight updates. We show that this perspective not only recovers model soup but enables more expressive and adaptive ensembling strategies. Empirically, AME produces averaged neural solutions that outperform both individual experts and model soup baselines, especially in out-of-distribution settings. Our results suggest a principled and generalizable notion of data-free model weight aggregation and defines, in one sense, how to perform neural averaging.