Generative Multiple-purpose Sampler for Weighted M-estimation

TL;DR

The paper introduces the Generative Multiple-purpose Sampler (GMS), a neural network-based method that efficiently produces solutions for weighted M-estimators, significantly reducing computational time for complex statistical procedures.

Contribution

It proposes a novel neural network architecture and a unified framework for fast, flexible weighted M-estimation, enabling procedures previously computationally infeasible.

Findings

GMS reduces computational time by orders of magnitude.

The weight multiplicative multilayer perceptron converges faster than traditional networks.

The R package GMS facilitates easy implementation of the proposed methods.

Abstract

To overcome the computational bottleneck of various data perturbation procedures such as the bootstrap and cross validations, we propose the Generative Multiple-purpose Sampler (GMS), which constructs a generator function to produce solutions of weighted M-estimators from a set of given weights and tuning parameters. The GMS is implemented by a single optimization without having to repeatedly evaluate the minimizers of weighted losses, and is thus capable of significantly reducing the computational time. We demonstrate that the GMS framework enables the implementation of various statistical procedures that would be unfeasible in a conventional framework, such as the iterated bootstrap, bootstrapped cross-validation for penalized likelihood, bootstrapped empirical Bayes with nonparametric maximum likelihood, etc. To construct a computationally efficient generator function, we also propose a novel form of neural network called the \emph{weight multiplicative multilayer perceptron} to achieve fast convergence. Our numerical results demonstrate that the new neural network structure enjoys a few orders of magnitude speed advantage in comparison to the conventional one. An R package called GMS is provided, which runs under Pytorch to implement the proposed methods and allows the user to provide a customized loss function to tailor to their own models of interest.