4Ward: a Relayering Strategy for Efficient Training of Arbitrarily Complex Directed Acyclic Graphs

TL;DR

4Ward is a Python library that efficiently generates and computes neural networks based on complex directed acyclic graphs, enabling scalable and customizable architectures beyond traditional layered models.

Contribution

The paper introduces 4Ward, a novel graph layering algorithm that efficiently constructs and computes complex neural networks with arbitrary DAG topologies, overcoming scalability and parallelization challenges.

Findings

Significant time gains in computational experiments with Erd"H{o}s-Rényi graphs.

Overcomes sequential computation limitations of previous methods.

Allows customization of weight initialization and activation functions.

Abstract

Thanks to their ease of implementation, multilayer perceptrons (MLPs) have become ubiquitous in deep learning applications. The graph underlying an MLP is indeed multipartite, i.e. each layer of neurons only connects to neurons belonging to the adjacent layer. In contrast, in vivo brain connectomes at the level of individual synapses suggest that biological neuronal networks are characterized by scale-free degree distributions or exponentially truncated power law strength distributions, hinting at potentially novel avenues for the exploitation of evolution-derived neuronal networks. In this paper, we present ``4Ward'', a method and Python library capable of generating flexible and efficient neural networks (NNs) from arbitrarily complex directed acyclic graphs. 4Ward is inspired by layering algorithms drawn from the graph drawing discipline to implement efficient forward passes, and provides significant time gains in computational experiments with various Erd\H{o}s-R\'enyi graphs. 4Ward not only overcomes the sequential nature of the learning matrix method, by parallelizing the computation of activations, but also addresses the scalability issues encountered in the current state-of-the-art and provides the designer with freedom to customize weight initialization and activation functions. Our algorithm can be of aid for any investigator seeking to exploit complex topologies in a NN design framework at the microscale.