Arax: A Runtime Framework for Decoupling Applications from Heterogeneous Accelerators

TL;DR

Arax is a runtime system that simplifies the use of heterogeneous accelerators by dynamically managing resources, enabling sharing, elasticity, and reducing programming effort with minimal overhead.

Contribution

Arax introduces a dynamic runtime framework that decouples applications from hardware accelerators, supporting resource sharing and elasticity with automatic stub generation.

Findings

Applications run with about 12% overhead using Arax.

Arax improves accelerator sharing, achieving up to 20% faster execution than NVIDIA MPS.

Elasticity support reduces total application turnaround time by up to 2x.

Abstract

Today, using multiple heterogeneous accelerators efficiently from applications and high-level frameworks, such as TensorFlow and Caffe, poses significant challenges in three respects: (a) sharing accelerators, (b) allocating available resources elastically during application execution, and (c) reducing the required programming effort. In this paper, we present Arax, a runtime system that decouples applications from heterogeneous accelerators within a server. First, Arax maps application tasks dynamically to available resources, managing all required task state, memory allocations, and task dependencies. As a result, Arax can share accelerators across applications in a server and adjust the resources used by each application as load fluctuates over time. dditionally, Arax offers a simple API and includes Autotalk, a stub generator that automatically generates stub libraries for applications already written for specific accelerator types, such as NVIDIA GPUs. Consequently, Arax applications are written once without considering physical details, including the number and type of accelerators. Our results show that applications, such as Caffe, TensorFlow, and Rodinia, can run using Arax with minimum effort and low overhead compared to native execution, about 12% (geometric mean). Arax supports efficient accelerator sharing, by offering up to 20% improved execution times compared to NVIDIA MPS, which supports NVIDIA GPUs only. Arax can transparently provide elasticity, decreasing total application turn-around time by up to 2x compared to native execution without elasticity support.