Parallel Algorithms Align with Neural Execution

TL;DR

This paper demonstrates that using parallel algorithms in neural network models aligns better with their inherent parallel processing capabilities, leading to faster training and improved predictive performance.

Contribution

It introduces parallel algorithm implementations for neural networks, showing significant reductions in training time and enhanced accuracy over sequential approaches.

Findings

Parallel algorithms reduce training time significantly.

Parallel implementations outperform sequential ones in accuracy.

Faster convergence observed with parallel methods.

Abstract

Neural algorithmic reasoners are parallel processors. Teaching them sequential algorithms contradicts this nature, rendering a significant share of their computations redundant. Parallel algorithms however may exploit their full computational power, therefore requiring fewer layers to be executed. This drastically reduces training times, as we observe when comparing parallel implementations of searching, sorting and finding strongly connected components to their sequential counterparts on the CLRS framework. Additionally, parallel versions achieve (often strongly) superior predictive performance.