Posterior Covariance Information Criterion for Weighted Inference

TL;DR

This paper introduces PCIC, a new information criterion for predictive evaluation in weighted inference scenarios, which generalizes WAIC and is computationally efficient with MCMC, applicable to covariate shift and counterfactual prediction.

Contribution

The paper develops PCIC, a novel information criterion that effectively handles weighted inference and generalizes WAIC for predictive scenarios with different likelihoods.

Findings

PCIC is asymptotically unbiased for quasi-Bayesian generalization error.

PCIC can be computed with a single MCMC run.

Numerical examples demonstrate practical applicability.

Abstract

For predictive evaluation based on quasi-posterior distributions, we develop a new information criterion, the posterior covariance information criterion (PCIC. PCIC generalises the widely applicable information criterion WAIC so as to effectively handle predictive scenarios where likelihoods for the estimation and the evaluation of the model may be different. A typical example of such scenarios is the weighted likelihood inference, including prediction under covariate shift and counterfactual prediction. The proposed criterion utilises a posterior covariance form and is computed by using only one Markov chain Monte Carlo run. Through numerical examples, we demonstrate how PCIC can apply in practice. Further, we show that PCIC is asymptotically unbiased to the quasi-Bayesian generalization error under mild conditions in weighted inference with both regular and singular statistical models.