Semigroups and sequential importance sampling for multiway tables and beyond

TL;DR

This paper improves sequential importance sampling for multiway tables by introducing a correction coefficient to reduce rejections, enhancing accuracy and efficiency in estimating table counts.

Contribution

It proposes a novel correction coefficient for SIS that asymptotically aligns the support with the marginal distribution, reducing rejections and bias.

Findings

The correction coefficient improves the accuracy of SIS in estimating table counts.

Extensive experiments show the correction reduces rejection rates.

Application to real data demonstrates practical effectiveness.

Abstract

When an interval of integers between the lower bound l_i and the upper bounds u_i is the support of the marginal distribution n_i|(n_{i-1}, ...,n_1), Chen et al. 2005 noticed that sampling from the interval at each step, for n_i during the sequential importance sampling (SIS) procedure, always produces a table which satisfies the marginal constraints. However, in general, the interval may not be equal to the support of the marginal distribution. In this case, the SIS procedure may produce tables which do not satisfy the marginal constraints, leading to rejection [Chen et al. 2006]. Rejecting tables is computationally expensive and incorrect proposal distributions result in biased estimators for the number of tables given its marginal sums. This paper has two focuses; (1) we propose a correction coefficient which corrects an interval of integers between the lower bound l_i and the upper bounds u_i to the support of the marginal distribution asymptotically even with rejections and with the same time complexity as the original SIS procedure (2) using univariate and bivariate logistic regression models, we present extensive experiments on simulated data sets for estimating the number of tables, and (3) we applied the volume test proposed by Diaconis and Efron 1985 on 2x2x6 randomly generated tables to compare the performance of SIS versus MCMC. When estimating the number of tables in our simulation study, we used univariate and bivariate logistic regression models since under these models the SIS procedure seems to have higher rate of rejections even with small tables. We also apply our correction coefficients to data sets on coronary heart disease and occurrence of esophageal cancer.