Deep Algorithms: designs for networks

TL;DR

This paper introduces a new neural network design methodology inspired by traditional algorithms, demonstrating how to incorporate weights and improve performance through training, matching traditional architectures.

Contribution

It presents a novel algorithmic-inspired design approach for neural networks, including heuristics and techniques for weight integration, with performance guarantees from initialization.

Findings

Networks initialized with known performance thresholds can be improved through training.

The proposed approach can match the performance of traditional neural network architectures.

Incorporating algorithmic design principles enhances neural network performance.

Abstract

A new design methodology for neural networks that is guided by traditional algorithm design is presented. To prove our point, we present two heuristics and demonstrate an algorithmic technique for incorporating additional weights in their signal-flow graphs. We show that with training the performance of these networks can not only exceed the performance of the initial network, but can match the performance of more-traditional neural network architectures. A key feature of our approach is that these networks are initialized with parameters that provide a known performance threshold for the architecture on a given task.