Graph-based Composite Local Bregman Divergences on Discrete Sample Spaces

TL;DR

This paper introduces a framework for statistical inference on discrete spaces using localized Bregman divergences, enabling efficient estimation with unnormalized models and applications to classification.

Contribution

It develops a general approach linking local scoring rules to localized Bregman divergences, analyzing their consistency and practical use in classification tasks.

Findings

Local scoring rules relate to localized Bregman divergences.

Consistency depends on neighborhood system structure.

Numerical experiments show the impact of neighborhood on accuracy.

Abstract

One of the most common methods for statistical inference is the maximum likelihood estimator (MLE). The MLE needs to compute the normalization constant in statistical models, and it is often intractable. Using unnormalized statistical models and replacing the likelihood with the other scoring rule are a good way to circumvent such high computation cost, where the scoring rule measures the goodness of fit of the model to observed samples. The scoring rule is closely related to the Bregman divergence, which is a discrepancy measure between two probability distributions. In this paper, the purpose is to provide a general framework of statistical inference using unnormalized statistical models on discrete sample spaces. A localized version of scoring rules is important to obtain computationally efficient estimators. We show that the local scoring rules are related to the localized version of Bregman divergences. Through the localized Bregman divergence, we investigate the statistical consistency of local scoring rules. We show that the consistency is determined by the structure of neighborhood system defined on discrete sample spaces. In addition, we show a way of applying local scoring rules to classification problems. In numerical experiments, we investigated the relation between the neighborhood system and the estimation accuracy.