Deep Generative Modeling on Limited Data with Regularization by Nontransferable Pre-trained Models

TL;DR

This paper introduces Reg-DGM, a regularization method for deep generative models that uses nontransferable pre-trained models to reduce variance and improve performance on limited data, supported by theoretical analysis and empirical results.

Contribution

It proposes a novel regularization framework for DGMs leveraging nontransferable pre-trained models, with theoretical guarantees and practical improvements.

Findings

Reg-DGM improves generative quality with limited data.

Theoretical analysis shows existence and convergence of the method.

Empirical results outperform state-of-the-art on various datasets.

Abstract

Deep generative models (DGMs) are data-eager because learning a complex model on limited data suffers from a large variance and easily overfits. Inspired by the classical perspective of the bias-variance tradeoff, we propose regularized deep generative model (Reg-DGM), which leverages a nontransferable pre-trained model to reduce the variance of generative modeling with limited data. Formally, Reg-DGM optimizes a weighted sum of a certain divergence and the expectation of an energy function, where the divergence is between the data and the model distributions, and the energy function is defined by the pre-trained model w.r.t. the model distribution. We analyze a simple yet representative Gaussian-fitting case to demonstrate how the weighting hyperparameter trades off the bias and the variance. Theoretically, we characterize the existence and the uniqueness of the global minimum of Reg-DGM in a non-parametric setting and prove its convergence with neural networks trained by gradient-based methods. Empirically, with various pre-trained feature extractors and a data-dependent energy function, Reg-DGM consistently improves the generation performance of strong DGMs with limited data and achieves competitive results to the state-of-the-art methods. Our implementation is available at https://github.com/ML-GSAI/Reg-ADA-APA.