Statistical guarantees for sparse deep learning

TL;DR

This paper develops statistical guarantees for various types of sparsity in deep neural networks, addressing multiple outputs, regularization, and l2-loss, thus enhancing theoretical understanding of wide and deep sparse models.

Contribution

It introduces new statistical guarantees for sparse deep learning considering different sparsity types and important aspects like multiple outputs and regularization.

Findings

Guarantees have mild dependence on network width and depth

Supports application of wide and deep sparse networks from a statistical perspective

Uses uncommon tools that may interest the deep learning research community

Abstract

Neural networks are becoming increasingly popular in applications, but our mathematical understanding of their potential and limitations is still limited. In this paper, we further this understanding by developing statistical guarantees for sparse deep learning. In contrast to previous work, we consider different types of sparsity, such as few active connections, few active nodes, and other norm-based types of sparsity. Moreover, our theories cover important aspects that previous theories have neglected, such as multiple outputs, regularization, and l2-loss. The guarantees have a mild dependence on network widths and depths, which means that they support the application of sparse but wide and deep networks from a statistical perspective. Some of the concepts and tools that we use in our derivations are uncommon in deep learning and, hence, might be of additional interest.