Graph Metanetworks for Processing Diverse Neural Architectures

TL;DR

This paper introduces Graph Metanetworks, a novel approach using graph neural networks to process and analyze diverse neural network architectures by representing their weights as graphs, enabling better generalization and symmetry handling.

Contribution

The paper presents Graph Metanetworks, a new method that generalizes to various neural architectures and handles symmetries, overcoming limitations of previous specialized approaches.

Findings

GMNs are expressive and permutation-equivariant.

Effective across multiple neural network architectures.

Outperform existing methods on metanetwork tasks.

Abstract

Neural networks efficiently encode learned information within their parameters. Consequently, many tasks can be unified by treating neural networks themselves as input data. When doing so, recent studies demonstrated the importance of accounting for the symmetries and geometry of parameter spaces. However, those works developed architectures tailored to specific networks such as MLPs and CNNs without normalization layers, and generalizing such architectures to other types of networks can be challenging. In this work, we overcome these challenges by building new metanetworks - neural networks that take weights from other neural networks as input. Put simply, we carefully build graphs representing the input neural networks and process the graphs using graph neural networks. Our approach, Graph Metanetworks (GMNs), generalizes to neural architectures where competing methods struggle, such as multi-head attention layers, normalization layers, convolutional layers, ResNet blocks, and group-equivariant linear layers. We prove that GMNs are expressive and equivariant to parameter permutation symmetries that leave the input neural network functions unchanged. We validate the effectiveness of our method on several metanetwork tasks over diverse neural network architectures.