Interpretation and Generalization of Score Matching

TL;DR

This paper explores the theoretical foundations of score matching, linking it to maximum likelihood, and extends its applicability to discrete data, enhancing robustness and versatility in high-dimensional density modeling.

Contribution

It establishes a formal connection between score matching and maximum likelihood, and introduces a generalization that enables application to discrete data models.

Findings

Score matching yields more robust parameters with noisy data.

A formal link between maximum likelihood and score matching is established.

Score matching is extended to models of discrete data.

Abstract

Score matching is a recently developed parameter learning method that is particularly effective to complicated high dimensional density models with intractable partition functions. In this paper, we study two issues that have not been completely resolved for score matching. First, we provide a formal link between maximum likelihood and score matching. Our analysis shows that score matching finds model parameters that are more robust with noisy training data. Second, we develop a generalization of score matching. Based on this generalization, we further demonstrate an extension of score matching to models of discrete data.