PairNets: Novel Fast Shallow Artificial Neural Networks on Partitioned Subspaces

TL;DR

This paper introduces PairNets, a fast, shallow neural network architecture that partitions input space into subspaces, enabling rapid training and improved performance over traditional ANNs in regression tasks.

Contribution

The paper presents a novel shallow 4-layer neural network called PairNet that uses space partitioning and linear system solving for quick hyperparameter optimization.

Findings

PairNets trained with one epoch outperform traditional ANNs in speed.

PairNets achieve lower mean squared errors in regression tasks.

Partitioned subspaces enhance training efficiency and accuracy.

Abstract

Traditionally, an artificial neural network (ANN) is trained slowly by a gradient descent algorithm such as the backpropagation algorithm since a large number of hyperparameters of the ANN need to be fine-tuned with many training epochs. To highly speed up training, we created a novel shallow 4-layer ANN called "Pairwise Neural Network" ("PairNet") with high-speed hyperparameter optimization. In addition, a value of each input is partitioned into multiple intervals, and then an n-dimensional space is partitioned into M n-dimensional subspaces. M local PairNets are built in M partitioned local n-dimensional subspaces. A local PairNet is trained very quickly with only one epoch since its hyperparameters are directly optimized one-time via simply solving a system of linear equations by using the multivariate least squares fitting method. Simulation results for three regression problems indicated that the PairNet achieved much higher speeds and lower average testing mean squared errors (MSEs) for the three cases, and lower average training MSEs for two cases than the traditional ANNs. A significant future work is to develop better and faster optimization algorithms based on intelligent methods and parallel computing methods to optimize both partitioned subspaces and hyperparameters to build the fast and effective PairNets for applications in big data mining and real-time machine learning.