Etalon: Holistic Performance Evaluation Framework for LLM Inference Systems

TL;DR

Etalon is a comprehensive framework for evaluating LLM inference systems, introducing a new fluidity-index metric to better capture real-time user experience, surpassing traditional latency and throughput metrics.

Contribution

The paper presents Etalon, a novel holistic evaluation framework with a new fluidity-index metric for assessing LLM inference systems' performance.

Findings

Current metrics inadequately assess real-time LLM inference performance

Etalon's fluidity-index provides deeper insights into user experience

Evaluation of open-source platforms reveals strengths and weaknesses

Abstract

Serving large language models (LLMs) in production can incur substantial costs, which has prompted recent advances in inference system optimizations. Today, these systems are evaluated against conventional latency and throughput metrics (eg. TTFT, TBT, Normalised Latency and TPOT). However, these metrics fail to fully capture the nuances of LLM inference, leading to an incomplete assessment of user-facing performance crucial for real-time applications such as chat and translation. In this paper, we first identify the pitfalls of current performance metrics in evaluating LLM inference systems. We then propose Etalon, a comprehensive performance evaluation framework that includes fluidity-index -- a novel metric designed to reflect the intricacies of the LLM inference process and its impact on real-time user experience. Finally, we evaluate various existing open-source platforms and model-as-a-service offerings using Etalon, discussing their strengths and weaknesses. Etalon is available at https://github.com/project-etalon/etalon.