Fast Compute via MC Boosting

TL;DR

This paper introduces Monte Carlo boosting as an efficient alternative for linear-system solves in statistical learning, combining random-walk estimators and residual correction to improve computational efficiency in repeated or partial solve scenarios.

Contribution

It surveys and unifies various Monte Carlo estimators and correction methods, extending them to overdetermined problems and connecting to existing algorithms like IRLS and EM.

Findings

Monte Carlo boosting scales well in certain regimes.

It outperforms traditional iterative methods in specific settings.

Provides practical guidance for integrating Monte Carlo methods into learning workflows.

Abstract

Modern training and inference pipelines in statistical learning and deep learning repeatedly invoke linear-system solves as inner loops, yet high-accuracy deterministic solvers can be prohibitively expensive when solves must be repeated many times or when only partial information (selected components or linear functionals) is required. We position \emph{Monte Carlo boosting} as a practical alternative in this regime, surveying random-walk estimators and sequential residual correction in a unified notation (Neumann-series representation, forward/adjoint estimators, and Halton-style sequential correction), with extensions to overdetermined/least-squares problems and connections to IRLS-style updates in data augmentation and EM/ECM algorithms. Empirically, we compare Jacobi and Gauss--Seidel iterations with plain Monte Carlo, exact sequential Monte Carlo, and a subsampled sequential variant, illustrating scaling regimes that motivate when Monte Carlo boosting can be an enabling compute primitive for modern statistical learning workflows.