On the H\"{o}lder Stability of Multiset and Graph Neural Networks

TL;DR

This paper introduces a new framework for analyzing the stability and separation quality of multiset and graph neural networks, addressing limitations of traditional separation measures and proposing improved models with better robustness.

Contribution

It proposes extit{ extbf{ extless}Holder extgreater{} in expectation}, a novel stability analysis framework, and develops new MPNNs with enhanced separation capabilities and robustness.

Findings

Common sum-based models are lower- extless{}Holder extgreater{} in expectation with depth

Adversarial graph examples can be separated by 1-WL iterations but not by standard MPNNs

New MPNNs with improved separation can classify adversarial examples and outperform standard models

Abstract

Extensive research efforts have been put into characterizing and constructing maximally separating multiset and graph neural networks. However, recent empirical evidence suggests the notion of separation itself doesn't capture several interesting phenomena. On the one hand, the quality of this separation may be very weak, to the extent that the embeddings of "separable" objects might even be considered identical when using fixed finite precision. On the other hand, architectures which aren't capable of separation in theory, somehow achieve separation when taking the network to be wide enough. In this work, we address both of these issues, by proposing a novel pair-wise separation quality analysis framework which is based on an adaptation of Lipschitz and \Holder{} stability to parametric functions. The proposed framework, which we name \emph{\Holder{} in expectation}, allows for separation quality analysis, without restricting the analysis to embeddings that can separate all the input space simultaneously. We prove that common sum-based models are lower-\Holder{} in expectation, with an exponent that decays rapidly with the network's depth . Our analysis leads to adversarial examples of graphs which can be separated by three 1-WL iterations, but cannot be separated in practice by standard maximally powerful Message Passing Neural Networks (MPNNs). To remedy this, we propose two novel MPNNs with improved separation quality, one of which is lower Lipschitz in expectation. We show these MPNNs can easily classify our adversarial examples, and compare favorably with standard MPNNs on standard graph learning tasks.