Non-Gradient Manifold Neural Network

TL;DR

This paper introduces a non-gradient optimization approach for neural networks that achieves rapid convergence and better data distribution modeling by using closed-form solutions and a novel decision layer.

Contribution

It presents a new manifold neural network based on non-gradient optimization, combining closed-form solutions with a manifold-aware decision layer.

Findings

Faster convergence compared to traditional gradient-based methods

Improved classification performance on benchmark datasets

Effective modeling of data distribution through the manifold decision layer

Abstract

Deep neural network (DNN) generally takes thousands of iterations to optimize via gradient descent and thus has a slow convergence. In addition, softmax, as a decision layer, may ignore the distribution information of the data during classification. Aiming to tackle the referred problems, we propose a novel manifold neural network based on non-gradient optimization, i.e., the closed-form solutions. Considering that the activation function is generally invertible, we reconstruct the network via forward ridge regression and low rank backward approximation, which achieve the rapid convergence. Moreover, by unifying the flexible Stiefel manifold and adaptive support vector machine, we devise the novel decision layer which efficiently fits the manifold structure of the data and label information. Consequently, a jointly non-gradient optimization method is designed to generate the network with closed-form results. Eventually, extensive experiments validate the superior performance of the model.