Efficient Random Sampling -- Parallel, Vectorized, Cache-Efficient, and Online

TL;DR

This paper introduces a simple, scalable, and cache-efficient divide-and-conquer algorithm for random sampling without replacement that performs well on modern parallel architectures, including GPUs and SIMD units.

Contribution

It presents a novel parallel and cache-efficient sampling algorithm with minimal communication, suitable for online and vectorized implementations on modern hardware.

Findings

Expected time complexity is O(n/p + log p) on p processors.

Minimal communication overhead of O(log p), independent of sample size.

Algorithm adapts to various sampling modes and hardware architectures.

Abstract

We consider the problem of sampling $n$ numbers from the range $\{1,\ldots,N\}$ without replacement on modern architectures. The main result is a simple divide-and-conquer scheme that makes sequential algorithms more cache efficient and leads to a parallel algorithm running in expected time $\mathcal{O}(n/p+\log p)$ on $p$ processors, i.e., scales to massively parallel machines even for moderate values of $n$. The amount of communication between the processors is very small (at most $\mathcal{O}(\log p)$) and independent of the sample size. We also discuss modifications needed for load balancing, online sampling, sampling with replacement, Bernoulli sampling, and vectorization on SIMD units or GPUs.