An Empirical Comparison of Sampling Quality Metrics: A Case Study for Bayesian Nonnegative Matrix Factorization

TL;DR

This paper empirically evaluates various metrics for assessing the quality of samples in Bayesian nonnegative matrix factorization, introducing new measures to better capture sampling diversity and the ability to explore multiple optima.

Contribution

It proposes new metrics for evaluating sampling diversity and explores their effectiveness in assessing Bayesian NMF sampling methods.

Findings

New metrics better capture sampling diversity.

Sampling methods vary in their ability to explore multiple optima.

Standard metrics may overlook important aspects of sampling quality.

Abstract

In this work, we empirically explore the question: how can we assess the quality of samples from some target distribution? We assume that the samples are provided by some valid Monte Carlo procedure, so we are guaranteed that the collection of samples will asymptotically approximate the true distribution. Most current evaluation approaches focus on two questions: (1) Has the chain mixed, that is, is it sampling from the distribution? and (2) How independent are the samples (as MCMC procedures produce correlated samples)? Focusing on the case of Bayesian nonnegative matrix factorization, we empirically evaluate standard metrics of sampler quality as well as propose new metrics to capture aspects that these measures fail to expose. The aspect of sampling that is of particular interest to us is the ability (or inability) of sampling methods to move between multiple optima in NMF problems. As a proxy, we propose and study a number of metrics that might quantify the diversity of a set of NMF factorizations obtained by a sampler through quantifying the coverage of the posterior distribution. We compare the performance of a number of standard sampling methods for NMF in terms of these new metrics.