On the Expressive Power of Permutation-Equivariant Weight-Space Networks

TL;DR

This paper develops a comprehensive theoretical framework to analyze the expressive power of permutation-equivariant weight-space neural networks, establishing their universality and equivalence under broad conditions.

Contribution

It provides the first unified theory characterizing the expressivity and universality of permutation-equivariant weight-space networks, clarifying their capabilities and limitations.

Findings

All prominent permutation-equivariant networks are equally expressive.

Universality holds under mild assumptions on input weights.

Edge cases where universality fails are characterized.

Abstract

Weight-space learning studies neural architectures that operate directly on the parameters of other neural networks. Motivated by the growing availability of pretrained models, recent work has demonstrated the effectiveness of weight-space networks across a wide range of tasks. SOTA weight-space networks rely on permutation-equivariant designs to improve generalization. However, this may negatively affect expressive power, warranting theoretical investigation. Importantly, unlike other structured domains, weight-space learning targets maps operating on both weight and function spaces, making expressivity analysis particularly subtle. While a few prior works provide partial expressivity results, a comprehensive characterization is still missing. In this work, we address this gap by developing a systematic theory for expressivity of weight-space networks. We first prove that all prominent permutation-equivariant networks are equivalent in expressive power. We then establish universality in both weight- and function-space settings under mild, natural assumptions on the input weights, and characterize the edge-case regimes where universality no longer holds. Together, these results provide a strong and unified foundation for the expressivity of weight-space networks.