Majorization Minimization Technique for Optimally Solving Deep Dictionary Learning

TL;DR

This paper introduces a novel Majorization Minimization-based method for simultaneously optimizing all layers in deep dictionary learning, leading to improved accuracy and speed over greedy approaches and other unsupervised deep learning models.

Contribution

It presents the first optimal, simultaneous solution for deep dictionary learning using Majorization Minimization, surpassing previous greedy methods.

Findings

Optimal deep dictionary learning improves accuracy.

Method outperforms other unsupervised models in speed.

Simultaneous optimization yields better representations.

Abstract

The concept of deep dictionary learning has been recently proposed. Unlike shallow dictionary learning which learns single level of dictionary to represent the data, it uses multiple layers of dictionaries. So far, the problem could only be solved in a greedy fashion; this was achieved by learning a single layer of dictionary in each stage where the coefficients from the previous layer acted as inputs to the subsequent layer (only the first layer used the training samples as inputs). This was not optimal; there was feedback from shallower to deeper layers but not the other way. This work proposes an optimal solution to deep dictionary learning whereby all the layers of dictionaries are solved simultaneously. We employ the Majorization Minimization approach. Experiments have been carried out on benchmark datasets; it shows that optimal learning indeed improves over greedy piecemeal learning. Comparison with other unsupervised deep learning tools (stacked denoising autoencoder, deep belief network, contractive autoencoder and K-sparse autoencoder) show that our method supersedes their performance both in accuracy and speed.