Exploiting the Statistics of Learning and Inference

TL;DR

This paper introduces algorithms that leverage the statistical properties of large datasets and simulations to enhance the efficiency of learning and inference, particularly through subsampling and uncertainty testing.

Contribution

It proposes novel methods for subsampling and uncertainty testing in stochastic gradients and MCMC, addressing computational challenges in big data and simulations.

Findings

Improved efficiency in stochastic gradient estimation.

Enhanced MCMC sampling using stochastic gradients.

Emphasizes the importance of storing all information in likelihood-free MCMC.

Abstract

When dealing with datasets containing a billion instances or with simulations that require a supercomputer to execute, computational resources become part of the equation. We can improve the efficiency of learning and inference by exploiting their inherent statistical nature. We propose algorithms that exploit the redundancy of data relative to a model by subsampling data-cases for every update and reasoning about the uncertainty created in this process. In the context of learning we propose to test for the probability that a stochastically estimated gradient points more than 180 degrees in the wrong direction. In the context of MCMC sampling we use stochastic gradients to improve the efficiency of MCMC updates, and hypothesis tests based on adaptive mini-batches to decide whether to accept or reject a proposed parameter update. Finally, we argue that in the context of likelihood free MCMC one needs to store all the information revealed by all simulations, for instance in a Gaussian process. We conclude that Bayesian methods will remain to play a crucial role in the era of big data and big simulations, but only if we overcome a number of computational challenges.