Mutual information estimation for graph convolutional neural networks

TL;DR

This paper introduces an architecture-agnostic method to estimate mutual information in neural networks, specifically applied to graph convolutional networks, to analyze internal representations and information flow during training.

Contribution

It presents a novel, architecture-agnostic approach for tracking mutual information in neural networks, with a focus on graph-based models, enhancing understanding of information flow.

Findings

Mutual information varies with network architecture and training.

Graph-based neural networks show distinct information flow patterns.

The method provides insights into how inductive biases affect internal representations.

Abstract

Measuring model performance is a key issue for deep learning practitioners. However, we often lack the ability to explain why a specific architecture attains superior predictive accuracy for a given data set. Often, validation accuracy is used as a performance heuristic quantifying how well a network generalizes to unseen data, but it does not capture anything about the information flow in the model. Mutual information can be used as a measure of the quality of internal representations in deep learning models, and the information plane may provide insights into whether the model exploits the available information in the data. The information plane has previously been explored for fully connected neural networks and convolutional architectures. We present an architecture-agnostic method for tracking a network's internal representations during training, which are then used to create the mutual information plane. The method is exemplified for graph-based neural networks fitted on citation data. We compare how the inductive bias introduced in graph-based architectures changes the mutual information plane relative to a fully connected neural network.