Misclassification bounds for PAC-Bayesian sparse deep learning

TL;DR

This paper develops PAC-Bayesian bounds for sparse deep learning with Spike-and-Slab priors, providing theoretical guarantees on misclassification errors and optimal model selection in high-dimensional classification tasks.

Contribution

It introduces novel PAC-Bayesian bounds for sparse deep neural networks with Spike-and-Slab priors, achieving minimax optimal rates and proposing an automated architecture selection method.

Findings

Non-asymptotic prediction error bounds established.

Achieves minimax optimal rates up to a logarithmic factor.

Proposes an automated model selection approach with guarantees.

Abstract

Recently, there has been a significant focus on exploring the theoretical aspects of deep learning, especially regarding its performance in classification tasks. Bayesian deep learning has emerged as a unified probabilistic framework, seeking to integrate deep learning with Bayesian methodologies seamlessly. However, there exists a gap in the theoretical understanding of Bayesian approaches in deep learning for classification. This study presents an attempt to bridge that gap. By leveraging PAC-Bayes bounds techniques, we present theoretical results on the prediction or misclassification error of a probabilistic approach utilizing Spike-and-Slab priors for sparse deep learning in classification. We establish non-asymptotic results for the prediction error. Additionally, we demonstrate that, by considering different architectures, our results can achieve minimax optimal rates in both low and high-dimensional settings, up to a logarithmic factor. Moreover, our additional logarithmic term yields slight improvements over previous works. Additionally, we propose and analyze an automated model selection approach aimed at optimally choosing a network architecture with guaranteed optimality.