PPI-SVRG: Unifying Prediction-Powered Inference and Variance Reduction for Semi-Supervised Optimization

TL;DR

This paper introduces PPI-SVRG, a semi-supervised optimization method that unifies prediction-powered inference and variance reduction, improving convergence and accuracy when labeled data is limited.

Contribution

It develops a novel algorithm combining PPI and SVRG, providing theoretical convergence guarantees and demonstrating significant empirical improvements.

Findings

Reduces MSE by 43-52% in mean estimation with limited labels

Improves test accuracy by 2.7-2.9% on MNIST with 10% labels

Theoretically decomposes convergence into standard rate plus prediction error

Abstract

We study semi-supervised stochastic optimization when labeled data is scarce but predictions from pre-trained models are available. PPI and SVRG both reduce variance through control variates -- PPI uses predictions, SVRG uses reference gradients. We show they are mathematically equivalent and develop PPI-SVRG, which combines both. Our convergence bound decomposes into the standard SVRG rate plus an error floor from prediction uncertainty. The rate depends only on loss geometry; predictions affect only the neighborhood size. When predictions are perfect, we recover SVRG exactly. When predictions degrade, convergence remains stable but reaches a larger neighborhood. Experiments confirm the theory: PPI-SVRG reduces MSE by 43--52\% under label scarcity on mean estimation benchmarks and improves test accuracy by 2.7--2.9 percentage points on MNIST with only 10\% labeled data.