Inconsistent parameter estimation in Markov random fields: Benefits in the computation-limited setting

TL;DR

This paper demonstrates that in computation-limited scenarios, using an intentionally inconsistent parameter estimator in Markov random fields can improve prediction accuracy by offsetting approximation errors, supported by theoretical and empirical analysis.

Contribution

It introduces a joint estimation and prediction method using convex variational relaxations, showing that inconsistency in parameter estimation can be beneficial in limited computation settings.

Findings

Inconsistent estimators can improve prediction accuracy in limited computation scenarios.

The proposed method outperforms traditional sum-product heuristics.

Theoretical analysis confirms stability and asymptotic properties of the estimators.

Abstract

Consider the problem of joint parameter estimation and prediction in a Markov random field: i.e., the model parameters are estimated on the basis of an initial set of data, and then the fitted model is used to perform prediction (e.g., smoothing, denoising, interpolation) on a new noisy observation. Working under the restriction of limited computation, we analyze a joint method in which the \emph{same convex variational relaxation} is used to construct an M-estimator for fitting parameters, and to perform approximate marginalization for the prediction step. The key result of this paper is that in the computation-limited setting, using an inconsistent parameter estimator (i.e., an estimator that returns the ``wrong'' model even in the infinite data limit) can be provably beneficial, since the resulting errors can partially compensate for errors made by using an approximate prediction technique. En route to this result, we analyze the asymptotic properties of M-estimators based on convex variational relaxations, and establish a Lipschitz stability property that holds for a broad class of variational methods. We show that joint estimation/prediction based on the reweighted sum-product algorithm substantially outperforms a commonly used heuristic based on ordinary sum-product.