Sparse Matrix Factorization

TL;DR

This paper introduces an algorithm for sparse matrix factorization that can recover network structures and hidden unit values in deep linear networks under certain conditions, linking it to deep learning and sparse recovery.

Contribution

It proposes a novel algorithm for sparse matrix factorization with theoretical guarantees for deep linear networks under sparsity and randomness assumptions.

Findings

Algorithm recovers network structure for depths up to  O(n^{1/6})

Proves recovery guarantees under specific sparsity assumptions

Links sparse matrix factorization to deep learning and dictionary learning

Abstract

We investigate the problem of factorizing a matrix into several sparse matrices and propose an algorithm for this under randomness and sparsity assumptions. This problem can be viewed as a simplification of the deep learning problem where finding a factorization corresponds to finding edges in different layers and values of hidden units. We prove that under certain assumptions for a sparse linear deep network with $n$ nodes in each layer, our algorithm is able to recover the structure of the network and values of top layer hidden units for depths up to $\tilde O(n^{1/6})$. We further discuss the relation among sparse matrix factorization, deep learning, sparse recovery and dictionary learning.