In All Likelihood, Deep Belief Is Not Enough

TL;DR

This paper introduces a practical estimator for likelihood in deep belief networks, enabling quantitative evaluation of their performance on natural images, and finds that such models may not be as effective as previously claimed.

Contribution

It presents a new, computationally feasible likelihood estimator for deep belief networks, allowing for rigorous quantitative assessment of these models.

Findings

The estimator is simple to apply and computationally tractable.

Deep belief networks may not be optimal for modeling natural images.

Quantitative analysis challenges earlier qualitative claims about model effectiveness.

Abstract

Statistical models of natural stimuli provide an important tool for researchers in the fields of machine learning and computational neuroscience. A canonical way to quantitatively assess and compare the performance of statistical models is given by the likelihood. One class of statistical models which has recently gained increasing popularity and has been applied to a variety of complex data are deep belief networks. Analyses of these models, however, have been typically limited to qualitative analyses based on samples due to the computationally intractable nature of the model likelihood. Motivated by these circumstances, the present article provides a consistent estimator for the likelihood that is both computationally tractable and simple to apply in practice. Using this estimator, a deep belief network which has been suggested for the modeling of natural image patches is quantitatively investigated and compared to other models of natural image patches. Contrary to earlier claims based on qualitative results, the results presented in this article provide evidence that the model under investigation is not a particularly good model for natural images