AI Tax: The Hidden Cost of AI Data Center Applications

TL;DR

This paper investigates the hidden costs of deploying AI workloads in edge data centers, revealing that AI acceleration can strain infrastructure and proposing a specialized design to reduce total cost of ownership.

Contribution

It highlights the overlooked infrastructure costs of AI acceleration and demonstrates a purpose-built edge data center design that lowers TCO by 15%.

Findings

AI workloads impose bottlenecks on storage and network bandwidth.

Specialized edge data center design reduces TCO by 15%.

Holistic analysis reveals infrastructure stresses not apparent in isolated AI algorithm evaluations.

Abstract

Artificial intelligence and machine learning are experiencing widespread adoption in industry and academia. This has been driven by rapid advances in the applications and accuracy of AI through increasingly complex algorithms and models; this, in turn, has spurred research into specialized hardware AI accelerators. Given the rapid pace of advances, it is easy to forget that they are often developed and evaluated in a vacuum without considering the full application environment. This paper emphasizes the need for a holistic, end-to-end analysis of AI workloads and reveals the "AI tax." We deploy and characterize Face Recognition in an edge data center. The application is an AI-centric edge video analytics application built using popular open source infrastructure and ML tools. Despite using state-of-the-art AI and ML algorithms, the application relies heavily on pre-and post-processing code. As AI-centric applications benefit from the acceleration promised by accelerators, we find they impose stresses on the hardware and software infrastructure: storage and network bandwidth become major bottlenecks with increasing AI acceleration. By specializing for AI applications, we show that a purpose-built edge data center can be designed for the stresses of accelerated AI at 15% lower TCO than one derived from homogeneous servers and infrastructure.