Benchmarking Compound AI Applications for Hardware-Software Co-Design

TL;DR

This paper introduces a benchmarking suite for Compound AI applications, enabling analysis of their complex deployment configurations to improve hardware-software co-design and resource efficiency.

Contribution

It presents a standardized benchmarking suite for Compound AI applications, facilitating cross-stack analysis and guiding system design improvements.

Findings

Derived key design principles for hardware-software co-design.

Identified factors influencing application performance and resource consumption.

Provided insights into optimizing deployment configurations.

Abstract

Compound AI applications, composed from interactions between Large Language Models (LLMs), Machine Learning (ML) models, external tools and data sources are quickly becoming an integral workload in datacenters. Their diverse sub-components and use-cases present a large configuration-space across the deployment stack -- ranging from applications and serving software down to hardware -- each of which may influence the application performance, deployment cost, and/or resource consumption. Despite their rapid adoption, however, the systems community lacks a standardized benchmark for analyzing this complicated design-space and guiding in system design. In this work, we present our benchmarking suite used for cross-stack analysis of Compound AI applications. Using this, we derive key takeaways and design principles spanning several layers of the stack for hardware-software co-design to unlock higher resource-efficiency.