Batch Matrix-form Equations and Implementation of Multilayer Perceptrons

TL;DR

This paper provides a comprehensive, mathematically rigorous batch matrix-form formulation of multilayer perceptrons, including forward and backward equations, validated with symbolic mathematics, and implements efficient, extensible reference code across multiple frameworks.

Contribution

It introduces a complete, explicit batch matrix-form derivation and implementation of MLPs, including advanced layers, validated with symbolic math, enabling transparent analysis and efficient sparse computation.

Findings

Validated all gradient equations using SymPy.

Developed uniform reference implementations in multiple frameworks.

Enabled efficient sparse neural network computations.

Abstract

Multilayer perceptrons (MLPs) remain fundamental to modern deep learning, yet their algorithmic details are rarely presented in complete, explicit \emph{batch matrix-form}. Rather, most references express gradients per sample or rely on automatic differentiation. Although automatic differentiation can achieve equally high computational efficiency, the usage of batch matrix-form makes the computational structure explicit, which is essential for transparent, systematic analysis, and optimization in settings such as sparse neural networks. This paper fills that gap by providing a mathematically rigorous and implementation-ready specification of MLPs in batch matrix-form. We derive forward and backward equations for all standard and advanced layers, including batch normalization and softmax, and validate all equations using the symbolic mathematics library SymPy. From these specifications, we construct uniform reference implementations in NumPy, PyTorch, JAX, TensorFlow, and a high-performance C++ backend optimized for sparse operations. Our main contributions are: (1) a complete derivation of batch matrix-form backpropagation for MLPs, (2) symbolic validation of all gradient equations, (3) uniform Python and C++ reference implementations grounded in a small set of matrix primitives, and (4) demonstration of how explicit formulations enable efficient sparse computation. Together, these results establish a validated, extensible foundation for understanding, teaching, and researching neural network algorithms.