Alternating optimization method based on nonnegative matrix factorizations for deep neural networks

TL;DR

This paper introduces a novel non-gradient-based method for training deep neural networks using semi-nonnegative matrix factorizations, avoiding backpropagation and achieving comparable error rates.

Contribution

It proposes an alternating optimization approach based on semi-NMFs for training DNNs, providing an alternative to backpropagation with similar accuracy.

Findings

Achieves comparable error rates to traditional backpropagation-based DNNs.

Demonstrates effectiveness of NMF-based pre-training for autoencoders.

Offers a gradient-free training method for deep neural networks.

Abstract

The backpropagation algorithm for calculating gradients has been widely used in computation of weights for deep neural networks (DNNs). This method requires derivatives of objective functions and has some difficulties finding appropriate parameters such as learning rate. In this paper, we propose a novel approach for computing weight matrices of fully-connected DNNs by using two types of semi-nonnegative matrix factorizations (semi-NMFs). In this method, optimization processes are performed by calculating weight matrices alternately, and backpropagation (BP) is not used. We also present a method to calculate stacked autoencoder using a NMF. The output results of the autoencoder are used as pre-training data for DNNs. The experimental results show that our method using three types of NMFs attains similar error rates to the conventional DNNs with BP.