Unified estimation framework for unnormalized models with statistical efficiency

TL;DR

This paper introduces a unified, statistically efficient estimation framework for unnormalized models that achieves the asymptotic efficiency of MLE with reasonable computational costs, applicable to both discrete and continuous sample spaces.

Contribution

The paper proposes a novel, unified estimation framework for unnormalized models that attains statistical efficiency comparable to MLE, addressing computational challenges and model misspecification.

Findings

The proposed estimators are asymptotically efficient, matching MLE variance.

The estimators are computationally feasible for both discrete and continuous data.

Experimental results show advantages over existing methods.

Abstract

The parameter estimation of unnormalized models is a challenging problem. The maximum likelihood estimation (MLE) is computationally infeasible for these models since normalizing constants are not explicitly calculated. Although some consistent estimators have been proposed earlier, the problem of statistical efficiency remains. In this study, we propose a unified, statistically efficient estimation framework for unnormalized models and several efficient estimators, whose asymptotic variance is the same as the MLE. The computational cost of these estimators is also reasonable and they can be employed whether the sample space is discrete or continuous. The loss functions of the proposed estimators are derived by combining the following two methods: (1) density-ratio matching using Bregman divergence, and (2) plugging-in nonparametric estimators. We also analyze the properties of the proposed estimators when the unnormalized models are misspecified. The experimental results demonstrate the advantages of our method over existing approaches.