Improving LLM Performance Through Black-Box Online Tuning: A Case for Adding System Specs to Factsheets for Trusted AI

TL;DR

This paper introduces a black-box online tuning method for LLMs that optimizes performance using only end-to-end measurements, emphasizing the importance of system specs in AI Factsheets for trustworthiness.

Contribution

It proposes a novel black-box controller for LLM performance tuning that does not require internal system access, enhancing AI system transparency and reliability.

Findings

Empirical evidence supports the effectiveness of the black-box controller.

Highlights the role of system specs in AI Factsheets for trust and sustainability.

Demonstrates improved LLM throughput through online tuning.

Abstract

In this paper, we present a novel black-box online controller that uses only end-to-end measurements over short segments, without internal instrumentation, and hill climbing to maximize goodput, defined as the throughput of requests that satisfy the service-level objective. We provide empirical evidence that this design is well-founded. Using this advance in LLM serving as a concrete example, we then discuss the importance of integrating system performance and sustainability metrics into Factsheets for organizations adopting AI systems.