Tensor-Based Backpropagation in Neural Networks with Non-Sequential Input

TL;DR

This paper explores tensor-based batch backpropagation algorithms that enable more efficient neural network training by leveraging tensor operations for non-linear backpropagation, reducing computational costs.

Contribution

It introduces a novel tensor-based approach to batch backpropagation that improves training efficiency over traditional methods.

Findings

Tensor operations enable non-linear backpropagation.

Training efficiency is increased with tensor-based algorithms.

Potential reduction in computational costs for neural network training.

Abstract

Neural networks have been able to achieve groundbreaking accuracy at tasks conventionally considered only doable by humans. Using stochastic gradient descent, optimization in many dimensions is made possible, albeit at a relatively high computational cost. By splitting training data into batches, networks can be distributed and trained vastly more efficiently and with minimal accuracy loss. We have explored the mathematics behind efficiently implementing tensor-based batch backpropagation algorithms. A common approach to batch training is iterating over batch items individually. Explicitly using tensor operations to backpropagate allows training to be performed non-linearly, increasing computational efficiency.