Accelerating Fast Fourier Transforms Using Hadoop and CUDA

TL;DR

This paper presents a novel approach combining Hadoop and CUDA to accelerate large-scale FFT processing by distributing workloads over GPU-equipped server clusters, significantly reducing processing time for massive files.

Contribution

It introduces a hybrid parallelization method leveraging Hadoop and CUDA to efficiently process terabyte-scale FFTs on commodity hardware clusters.

Findings

Achieved faster FFT processing on large files using GPU clusters

Demonstrated cost-effective performance on Amazon EC2

Outperformed traditional supercomputers in specific benchmarks

Abstract

There has been considerable research into improving Fast Fourier Transform (FFT) performance through parallelization and optimization for specialized hardware. However, even with those advancements, processing of very large files, over 1TB in size, still remains prohibitively slow. Analysts performing signal processing are forced to wait hours or days for results, which results in a disruption of their workflow and a decrease in productivity. In this paper we present a unique approach that not only parallelizes the workload over multi-cores, but distributes the problem over a cluster of graphics processing unit (GPU)-equipped servers. By utilizing Hadoop and CUDA, we can take advantage of inexpensive servers while still exceeding the processing power of a dedicated supercomputer, as demonstrated in our result using Amazon EC2.