Machine Learning for CUDA+MPI Design Rules

TL;DR

This paper introduces a machine learning-based approach to automatically explore and generate design rules for CUDA+MPI programs, helping optimize performance by identifying impactful design choices.

Contribution

It presents a novel method combining Monte-Carlo tree search and decision trees to efficiently discover and recommend high-impact design configurations for CUDA+MPI applications.

Findings

Effective identification of performance-critical design regions

Automated generation of design rules for CUDA+MPI programs

Demonstrated on sparse-matrix vector multiplication

Abstract

We present a new strategy for automatically exploring the design space of key CUDA+MPI programs and providing design rules that discriminate slow from fast implementations. In such programs, the order of operations (e.g., GPU kernels, MPI communication) and assignment of operations to resources (e.g., GPU streams) makes the space of possible designs enormous. Systems experts have the task of redesigning and reoptimizing these programs to effectively utilize each new platform. This work provides a prototype tool to reduce that burden. In our approach, a directed acyclic graph of CUDA and MPI operations defines the design space for the program. Monte-Carlo tree search discovers regions of the design space that have large impact on the program's performance. A sequence-to-vector transformation defines features for each explored implementation, and each implementation is assigned a class label according to its relative performance. A decision tree is trained on the features and labels to produce design rules for each class; these rules can be used by systems experts to guide their implementations. We demonstrate our strategy using a key kernel from scientific computing -- sparse-matrix vector multiplication -- on a platform with multiple MPI ranks and GPU streams.