What makes a good feedforward computational graph?

TL;DR

This paper investigates the properties of feedforward computational graphs in neural networks, introducing measures like fidelity and mixing time, and evaluates how these influence model performance through theoretical and empirical analyses.

Contribution

It introduces and analyzes two key measures for feedforward graphs, fidelity and mixing time, providing insights into their impact on neural network performance.

Findings

Fidelity and mixing time are critical metrics for feedforward graph quality.

Certain graph structures outperform others based on these measures.

Theoretical analysis aligns with empirical performance results.

Abstract

As implied by the plethora of literature on graph rewiring, the choice of computational graph employed by a neural network can make a significant impact on its downstream performance. Certain effects related to the computational graph, such as under-reaching and over-squashing, may even render the model incapable of learning certain functions. Most of these effects have only been thoroughly studied in the domain of undirected graphs; however, recent years have seen a significant rise in interest in feedforward computational graphs: directed graphs without any back edges. In this paper, we study the desirable properties of a feedforward computational graph, discovering two important complementary measures: fidelity and mixing time, and evaluating a few popular choices of graphs through the lens of these measures. Our study is backed by both theoretical analyses of the metrics' asymptotic behaviour for various graphs, as well as correlating these metrics to the performance of trained neural network models using the corresponding graphs.