Quasi-Equivariant Metanetworks

TL;DR

This paper introduces quasi-equivariance in metanetworks, allowing more flexible symmetry preservation in neural architectures to improve expressivity and robustness.

Contribution

It proposes the concept of quasi-equivariance, extending strict equivariance to better preserve functional identity while enhancing model expressiveness.

Findings

Quasi-equivariant metanetworks outperform strictly equivariant models in expressivity.

The framework applies broadly across feedforward, convolutional, and transformer architectures.

Empirical results show improved trade-offs between symmetry and expressiveness.

Abstract

Metanetworks are neural architectures designed to operate directly on pretrained weights to perform downstream tasks. However, the parameter space serves only as a proxy for the underlying function class, and the parameter-function mapping is inherently non-injective: distinct parameter configurations may yield identical input-output behaviors. As a result, metanetworks that rely solely on raw parameters risk overlooking the intrinsic symmetries of the architecture. Reasoning about functional identity is therefore essential for effective metanetwork design, motivating the development of equivariant metanetworks, which incorporate equivariance principles to respect architectural symmetries. Existing approaches, however, typically enforce strict equivariance, which imposes rigid constraints and often leads to sparse and less expressive models. To address this limitation, we introduce the novel concept of quasi-equivariance, which allows metanetworks to move beyond the rigidity of strict equivariance while still preserving functional identity. We lay down a principled basis for this framework and demonstrate its broad applicability across diverse neural architectures, including feedforward, convolutional, and transformer networks. Through empirical evaluation, we show that quasi-equivariant metanetworks achieve good trade-offs between symmetry preservation and representational expressivity. These findings advance the theoretical understanding of weight-space learning and provide a principled foundation for the design of more expressive and functionally robust metanetworks.