Horn: A System for Parallel Training and Regularizing of Large-Scale Neural Networks

TL;DR

Horn is a distributed system designed to efficiently train and regularize large neural networks using flexible partitioning and parallelization strategies, demonstrated on MNIST classification.

Contribution

The paper introduces Horn, a novel distributed system that enables effective training and regularization of large-scale neural networks with neuron-centric computation and parallel dropout.

Findings

Effective model partitioning improves training efficiency.

Parallel dropout enhances regularization in distributed settings.

Successful experiments on MNIST demonstrate system viability.

Abstract

I introduce a new distributed system for effective training and regularizing of Large-Scale Neural Networks on distributed computing architectures. The experiments demonstrate the effectiveness of flexible model partitioning and parallelization strategies based on neuron-centric computation model, with an implementation of the collective and parallel dropout neural networks training. Experiments are performed on MNIST handwritten digits classification including results.