Software Performance Engineering for Foundation Model-Powered Software

TL;DR

This paper emphasizes the importance of Software Performance Engineering in Foundation Model-based software, addressing challenges in design, communication, tuning, and deployment to ensure efficiency and performance.

Contribution

It identifies key challenges in performance engineering for FMware and discusses innovative solutions based on literature and practical experience.

Findings

Performance engineering is often overlooked in FMware development.

Four key challenges are identified: architecture, communication, tuning, deployment.

Continuous performance monitoring is crucial for FMware efficiency.

Abstract

The rise of Foundation Models (FMs) like Large Language Models (LLMs) is revolutionizing software development. Despite the impressive prototypes, transforming FMware into production-ready products demands complex engineering across various domains. A critical but overlooked aspect is performance engineering, which aims at ensuring FMware meets performance goals such as throughput and latency to avoid user dissatisfaction and financial loss. Often, performance considerations are an afterthought, leading to costly optimization efforts post-deployment. FMware's high computational resource demands highlight the need for efficient hardware use. Continuous performance engineering is essential to prevent degradation. This paper highlights the significance of Software Performance Engineering (SPE) in FMware, identifying four key challenges: cognitive architecture design (i.e., the structural design that defines how AI components interact, reason, and interface with classical software components), communication protocols, tuning and optimization, and deployment. These challenges are based on literature surveys and experiences from developing an in-house FMware system. We discuss problems, current practices, and innovative paths for the software engineering community.