Relating graph auto-encoders to linear models

TL;DR

This paper proves that graph auto-encoders have a subset of the solution space of linear models, and demonstrates that linear models can outperform nonlinear auto-encoders when leveraging node features, challenging assumptions about nonlinearity's importance.

Contribution

The work establishes the theoretical relationship between graph auto-encoders and linear models, highlighting the significance of node features as an inductive bias.

Findings

Linear models have at least the same representational power as graph auto-encoders.

Linear encoders can outperform nonlinear ones when using node features.

The node features serve as a more influential inductive bias than nonlinearity.

Abstract

Graph auto-encoders are widely used to construct graph representations in Euclidean vector spaces. However, it has already been pointed out empirically that linear models on many tasks can outperform graph auto-encoders. In our work, we prove that the solution space induced by graph auto-encoders is a subset of the solution space of a linear map. This demonstrates that linear embedding models have at least the representational power of graph auto-encoders based on graph convolutional networks. So why are we still using nonlinear graph auto-encoders? One reason could be that actively restricting the linear solution space might introduce an inductive bias that helps improve learning and generalization. While many researchers believe that the nonlinearity of the encoder is the critical ingredient towards this end, we instead identify the node features of the graph as a more powerful inductive bias. We give theoretical insights by introducing a corresponding bias in a linear model and analyzing the change in the solution space. Our experiments are aligned with other empirical work on this question and show that the linear encoder can outperform the nonlinear encoder when using feature information.