When the whole is greater than the sum of its parts: Scaling black-box inference to large data settings through divide-and-conquer

TL;DR

This paper introduces a divide-and-conquer framework for black-box inference in large, complex datasets, significantly reducing computational costs by partitioning data, parallel processing, and combining estimates efficiently.

Contribution

It presents a novel scalable approach for black-box inference that handles expensive data simulation, enabling analysis of large spatial datasets with improved efficiency.

Findings

Accelerates inference by partitioning data and parallel processing.

Enables analysis of large-scale spatial data with tens of thousands of locations.

Demonstrates effectiveness on climate and observational temperature data.

Abstract

Black-box methods such as deep neural networks are exceptionally fast at obtaining point estimates of model parameters due to their amortisation of the loss function computation, but are currently restricted to settings for which simulating training data is inexpensive. When simulating data is computationally expensive, both the training and uncertainty quantification, which typically relies on a parametric bootstrap, become intractable. We propose a black-box divide-and-conquer estimation and inference framework when data simulation is computationally expensive that trains a black-box estimation method on a partition of the multivariate data domain, estimates and bootstraps on the partitioned data, and combines estimates and inferences across data partitions. Through the divide step, only small training data need be simulated, substantially accelerating the training. Further, the estimation and bootstrapping can be conducted in parallel across multiple computing nodes to further speed up the procedure. Finally, the conquer step accounts for any dependence between data partitions through a statistically and computationally efficient weighted average. We illustrate the implementation of our framework in high-dimensional spatial settings with Gaussian and max-stable processes. Applications to modeling extremal temperature data from both a climate model and observations from the National Oceanic and Atmospheric Administration highlight the feasibility of estimation and inference of max-stable process parameters with tens of thousands of locations.