Optimal regularizations for data generation with probabilistic graphical models

TL;DR

This paper analyzes how optimal L1 and L2 regularizations improve the quality of generative models in graphical models by balancing likelihoods of training, test, and generated data, revealing a universal optimal regularization strength.

Contribution

It provides analytical and numerical evidence that the optimal regularization strength is approximately the inverse of the sum of squared couplings, largely independent of network structure.

Findings

Optimal regularization strength is about the inverse sum of squared couplings.

Test and generated data likelihoods are close at optimal regularization.

Results are robust across different interaction structures and regularization schemes.

Abstract

Understanding the role of regularization is a central question in Statistical Inference. Empirically, well-chosen regularization schemes often dramatically improve the quality of the inferred models by avoiding overfitting of the training data. We consider here the particular case of L 2 and L 1 regularizations in the Maximum A Posteriori (MAP) inference of generative pairwise graphical models. Based on analytical calculations on Gaussian multivariate distributions and numerical experiments on Gaussian and Potts models we study the likelihoods of the training, test, and 'generated data' (with the inferred models) sets as functions of the regularization strengths. We show in particular that, at its maximum, the test likelihood and the 'generated' likelihood, which quantifies the quality of the generated samples, have remarkably close values. The optimal value for the regularization strength is found to be approximately equal to the inverse sum of the squared couplings incoming on sites on the underlying network of interactions. Our results seem largely independent of the structure of the true underlying interactions that generated the data, of the regularization scheme considered, and are valid when small fluctuations of the posterior distribution around the MAP estimator are taken into account. Connections with empirical works on protein models learned from homologous sequences are discussed.