A Blueprint Architecture of Compound AI Systems for Enterprise

TL;DR

This paper proposes a blueprint architecture for enterprise compound AI systems that integrates LLMs with existing infrastructure, emphasizing cost-effectiveness, seamless operation, and coordination among components using stream orchestration.

Contribution

It introduces a novel blueprint architecture for compound AI systems in enterprise settings, focusing on seamless integration, orchestration, and task/data planning.

Findings

Efficient integration of LLMs with enterprise infrastructure.

Stream orchestration for coordinating AI components.

Task and data planning for optimized AI workflows.

Abstract

Large Language Models (LLMs) have showcased remarkable capabilities surpassing conventional NLP challenges, creating opportunities for use in production use cases. Towards this goal, there is a notable shift to building compound AI systems, wherein LLMs are integrated into an expansive software infrastructure with many components like models, retrievers, databases and tools. In this paper, we introduce a blueprint architecture for compound AI systems to operate in enterprise settings cost-effectively and feasibly. Our proposed architecture aims for seamless integration with existing compute and data infrastructure, with ``stream'' serving as the key orchestration concept to coordinate data and instructions among agents and other components. Task and data planners, respectively, break down, map, and optimize tasks and data to available agents and data sources defined in respective registries, given production constraints such as accuracy and latency.