GPUs, CPUs, and... NICs: Rethinking the Network's Role in Serving Complex AI Pipelines

TL;DR

This paper explores leveraging network hardware, particularly SmartNICs, to offload data processing tasks in AI pipelines, aiming to reduce resource overheads and mitigate network delays in distributed AI inference systems.

Contribution

It proposes integrating network hardware like SmartNICs into AI pipelines to offload data processing, offering a new approach to optimize distributed AI serving.

Findings

Offloading data processing to SmartNICs can reduce resource overheads.

Network hardware integration offers new optimization opportunities for AI pipelines.

Challenges include hardware compatibility and task offloading strategies.

Abstract

The increasing prominence of AI necessitates the deployment of inference platforms for efficient and effective management of AI pipelines and compute resources. As these pipelines grow in complexity, the demand for distributed serving rises and introduces much-dreaded network delays. In this paper, we investigate how the network can instead be a boon to the excessively high resource overheads of AI pipelines. To alleviate these overheads, we discuss how resource-intensive data processing tasks -- a key facet of growing AI pipeline complexity -- are well-matched for the computational characteristics of packet processing pipelines and how they can be offloaded onto SmartNICs. We explore the challenges and opportunities of offloading, and propose a research agenda for integrating network hardware into AI pipelines, unlocking new opportunities for optimization.