# Data Augmentation for Bayesian Deep Learning

## TL;DR

This paper introduces data augmentation techniques for Bayesian deep learning that improve uncertainty quantification and training efficiency by leveraging scale mixture models and advanced algorithms.

## Contribution

It develops novel data augmentation strategies for deep learning using scale mixture theory, enabling scalable EM and MCMC methods for uncertainty quantification.

## Key findings

- Enhanced training efficiency over traditional SGD
- Effective uncertainty quantification in deep learning models
- Scalable algorithms for high-dimensional models

## Abstract

Deep Learning (DL) methods have emerged as one of the most powerful tools for functional approximation and prediction. While the representation properties of DL have been well studied, uncertainty quantification remains challenging and largely unexplored. Data augmentation techniques are a natural approach to provide uncertainty quantification and to incorporate stochastic Monte Carlo search into stochastic gradient descent (SGD) methods. The purpose of our paper is to show that training DL architectures with data augmentation leads to efficiency gains. We use the theory of scale mixtures of normals to derive data augmentation strategies for deep learning. This allows variants of the expectation-maximization and MCMC algorithms to be brought to bear on these high dimensional nonlinear deep learning models. To demonstrate our methodology, we develop data augmentation algorithms for a variety of commonly used activation functions: logit, ReLU, leaky ReLU and SVM. Our methodology is compared to traditional stochastic gradient descent with back-propagation. Our optimization procedure leads to a version of iteratively re-weighted least squares and can be implemented at scale with accelerated linear algebra methods providing substantial improvement in speed. We illustrate our methodology on a number of standard datasets. Finally, we conclude with directions for future research.