SwinGNN: Rethinking Permutation Invariance in Diffusion Models for Graph Generation

TL;DR

SwinGNN introduces a non-invariant diffusion model for graph generation that leverages shifted window self-attention and effective training techniques, achieving state-of-the-art results on various datasets.

Contribution

The paper proposes a novel non-invariant diffusion model, SwinGNN, with a unique message passing network and training strategies, surpassing existing permutation-invariant models.

Findings

SwinGNN outperforms previous models on synthetic and real-world datasets.

The proposed post-processing improves permutation invariance of generated graphs.

Effective training and sampling techniques significantly enhance sample quality.

Abstract

Diffusion models based on permutation-equivariant networks can learn permutation-invariant distributions for graph data. However, in comparison to their non-invariant counterparts, we have found that these invariant models encounter greater learning challenges since 1) their effective target distributions exhibit more modes; 2) their optimal one-step denoising scores are the score functions of Gaussian mixtures with more components. Motivated by this analysis, we propose a non-invariant diffusion model, called $\textit{SwinGNN}$, which employs an efficient edge-to-edge 2-WL message passing network and utilizes shifted window based self-attention inspired by SwinTransformers. Further, through systematic ablations, we identify several critical training and sampling techniques that significantly improve the sample quality of graph generation. At last, we introduce a simple post-processing trick, $\textit{i.e.}$, randomly permuting the generated graphs, which provably converts any graph generative model to a permutation-invariant one. Extensive experiments on synthetic and real-world protein and molecule datasets show that our SwinGNN achieves state-of-the-art performances. Our code is released at https://github.com/qiyan98/SwinGNN.