Asymptotic Efficiency of Deterministic Estimators for Discrete Energy-Based Models: Ratio Matching and Pseudolikelihood

TL;DR

This paper analyzes the asymptotic efficiency of deterministic estimators like pseudolikelihood and ratio matching for discrete energy-based models, providing theoretical insights into their statistical properties.

Contribution

It introduces a generalized estimator framework and derives its asymptotic covariance, comparing the efficiency of existing estimators.

Findings

Pseudolikelihood and ratio matching estimators have quantifiable asymptotic efficiencies.

Theoretical expressions for the asymptotic covariance matrix are provided.

The study offers insights into the relative performance of these estimators.

Abstract

Standard maximum likelihood estimation cannot be applied to discrete energy-based models in the general case because the computation of exact model probabilities is intractable. Recent research has seen the proposal of several new estimators designed specifically to overcome this intractability, but virtually nothing is known about their theoretical properties. In this paper, we present a generalized estimator that unifies many of the classical and recently proposed estimators. We use results from the standard asymptotic theory for M-estimators to derive a generic expression for the asymptotic covariance matrix of our generalized estimator. We apply these results to study the relative statistical efficiency of classical pseudolikelihood and the recently-proposed ratio matching estimator.