Non-iterative recomputation of dense layers for performance improvement of DCNN

TL;DR

This paper introduces a non-iterative learning method for DCNNs that replaces backpropagation in dense layers with Moore-Penrose Inverse computations, leading to faster training and improved performance across multiple benchmarks.

Contribution

It proposes a novel non-iterative training strategy for dense layers in DCNNs using Moore-Penrose Inverse, reducing training time and enhancing accuracy.

Findings

Significant performance improvements over 30 state-of-the-art methods.

Faster training process compared to traditional backpropagation.

Better generalization with the proposed recomputation approach.

Abstract

An iterative method of learning has become a paradigm for training deep convolutional neural networks (DCNN). However, utilizing a non-iterative learning strategy can accelerate the training process of the DCNN and surprisingly such approach has been rarely explored by the deep learning (DL) community. It motivates this paper to introduce a non-iterative learning strategy that eliminates the backpropagation (BP) at the top dense or fully connected (FC) layers of DCNN, resulting in, lower training time and higher performance. The proposed method exploits the Moore-Penrose Inverse to pull back the current residual error to each FC layer, generating well-generalized features. Then using the recomputed features, i.e., the new generalized features the weights of each FC layer is computed according to the Moore-Penrose Inverse. We evaluate the proposed approach on six widely accepted object recognition benchmark datasets: Scene-15, CIFAR-10, CIFAR-100, SUN-397, Places365, and ImageNet. The experimental results show that the proposed method obtains significant improvements over 30 state-of-the-art methods. Interestingly, it also indicates that any DCNN with the proposed method can provide better performance than the same network with its original training based on BP.