# U-Statistics for Importance-Weighted Variational Inference

## TL;DR

This paper introduces U-statistics to importance-weighted variational inference, significantly reducing gradient estimator variance and improving inference performance with minimal additional computational cost.

## Contribution

It applies U-statistics to variational inference, providing a novel variance reduction technique with theoretical analysis and practical efficiency.

## Key findings

- U-statistics reduce variance of gradient estimates.
- Empirical improvements in inference performance.
- Minimal additional computational cost.

## Abstract

We propose the use of U-statistics to reduce variance for gradient estimation in importance-weighted variational inference. The key observation is that, given a base gradient estimator that requires $m > 1$ samples and a total of $n > m$ samples to be used for estimation, lower variance is achieved by averaging the base estimator on overlapping batches of size $m$ than disjoint batches, as currently done. We use classical U-statistic theory to analyze the variance reduction, and propose novel approximations with theoretical guarantees to ensure computational efficiency. We find empirically that U-statistic variance reduction can lead to modest to significant improvements in inference performance on a range of models, with little computational cost.