DaggerFFT: A Distributed FFT Framework Using Task Scheduling in Julia

TL;DR

DaggerFFT is a Julia-based distributed FFT framework that uses dynamic task scheduling to improve scalability and performance on heterogeneous HPC systems, outperforming existing libraries.

Contribution

Introduces DaggerFFT, a novel distributed FFT framework leveraging dynamic task scheduling in Julia, enhancing scalability and performance on CPU and GPU clusters.

Findings

Achieves up to 2.6x speedup on CPU clusters

Achieves up to 1.35x speedup on GPU clusters

Demonstrates improved performance and modularity in real-world simulations

Abstract

The Fast Fourier Transform (FFT) is a fundamental numerical technique with widespread application in a range of scientific problems. As scientific simulations attempt to exploit exascale systems, there has been a growing demand for distributed FFT algorithms that can effectively utilize modern heterogeneous high-performance computing (HPC) systems. Conventional FFT algorithms commonly encounter performance bottlenecks, especially when run on heterogeneous platforms. Most distributed FFT approaches rely on static task distribution and require synchronization barriers, limiting scalability and impacting overall resource utilization. In this paper we present DaggerFFT, a distributed FFT framework, developed in Julia, that treats highly parallel FFT computations as a dynamically scheduled task graph. Each FFT stage operates on a separately defined distributed array. FFT operations are expressed as DTasks operating on pencil or slab partitioned DArrays. Each FFT stage owns its own DArray, and the runtime assigns DTasks across devices using Dagger's dynamic scheduler that uses work stealing. We demonstrate how DaggerFFT's dynamic scheduler can outperform state-of-the-art distributed FFT libraries on both CPU and GPU backends, achieving up to a 2.6x speedup on CPU clusters and up to a 1.35x speedup on GPU clusters. We have integrated DaggerFFT into Oceananigans.jl, a geophysical fluid dynamics framework, demonstrating that high-level, task-based runtimes can deliver both superior performance and modularity in large-scale, real-world simulations.