Graphics Processing Units and High-Dimensional Optimization

TL;DR

This paper explores how graphics processing units (GPUs) can significantly accelerate high-dimensional statistical algorithms, demonstrating speedups of up to 100 times and emphasizing their transformative potential in computational statistics.

Contribution

It highlights the suitability of certain optimization algorithms for GPU acceleration and demonstrates practical applications with substantial speed improvements.

Findings

GPUs enable up to 100-fold speedups in statistical computations.

Algorithms like EM and MM are well-suited for GPU parallelization.

GPU technology is poised to revolutionize computational statistics in the coming decade.

Abstract

This paper discusses the potential of graphics processing units (GPUs) in high-dimensional optimization problems. A single GPU card with hundreds of arithmetic cores can be inserted in a personal computer and dramatically accelerates many statistical algorithms. To exploit these devices fully, optimization algorithms should reduce to multiple parallel tasks, each accessing a limited amount of data. These criteria favor EM and MM algorithms that separate parameters and data. To a lesser extent block relaxation and coordinate descent and ascent also qualify. We demonstrate the utility of GPUs in nonnegative matrix factorization, PET image reconstruction, and multidimensional scaling. Speedups of 100 fold can easily be attained. Over the next decade, GPUs will fundamentally alter the landscape of computational statistics. It is time for more statisticians to get on-board.