Learning Stochastic Dynamical Systems as an Implicit Regularization with Graph Neural Networks

TL;DR

This paper introduces stochastic Gumbel graph networks (S-GGNs) that learn high-dimensional time series with spatial correlations, revealing the implicit regularization effects of noise and demonstrating superior performance on real-world data.

Contribution

The paper proposes a novel S-GGNs framework that captures randomness and spatial correlations via Gumbel embeddings, with theoretical guarantees and empirical validation.

Findings

S-GGNs show improved convergence and robustness

Theoretical analysis of loss function differences

Superior generalization on real-world data

Abstract

Stochastic Gumbel graph networks are proposed to learn high-dimensional time series, where the observed dimensions are often spatially correlated. To that end, the observed randomness and spatial-correlations are captured by learning the drift and diffusion terms of the stochastic differential equation with a Gumble matrix embedding, respectively. In particular, this novel framework enables us to investigate the implicit regularization effect of the noise terms in S-GGNs. We provide a theoretical guarantee for the proposed S-GGNs by deriving the difference between the two corresponding loss functions in a small neighborhood of weight. Then, we employ Kuramoto's model to generate data for comparing the spectral density from the Hessian Matrix of the two loss functions. Experimental results on real-world data, demonstrate that S-GGNs exhibit superior convergence, robustness, and generalization, compared with state-of-the-arts.