GPUs, CPUs, and... NICs: Rethinking the Network's Role in Serving Complex AI Pipelines
Mike Wong, Ulysses Butler, Emma Farkash, Praveen Tammana, Anirudh, Sivaraman, Ravi Netravali

TL;DR
This paper explores leveraging network hardware, particularly SmartNICs, to offload data processing tasks in AI pipelines, aiming to reduce resource overheads and mitigate network delays in distributed AI inference systems.
Contribution
It proposes integrating network hardware like SmartNICs into AI pipelines to offload data processing, offering a new approach to optimize distributed AI serving.
Findings
Offloading data processing to SmartNICs can reduce resource overheads.
Network hardware integration offers new optimization opportunities for AI pipelines.
Challenges include hardware compatibility and task offloading strategies.
Abstract
The increasing prominence of AI necessitates the deployment of inference platforms for efficient and effective management of AI pipelines and compute resources. As these pipelines grow in complexity, the demand for distributed serving rises and introduces much-dreaded network delays. In this paper, we investigate how the network can instead be a boon to the excessively high resource overheads of AI pipelines. To alleviate these overheads, we discuss how resource-intensive data processing tasks -- a key facet of growing AI pipeline complexity -- are well-matched for the computational characteristics of packet processing pipelines and how they can be offloaded onto SmartNICs. We explore the challenges and opportunities of offloading, and propose a research agenda for integrating network hardware into AI pipelines, unlocking new opportunities for optimization.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsBrain Tumor Detection and Classification · IoT and Edge/Fog Computing · Big Data and Digital Economy
