PromptTuner: SLO-Aware Elastic System for LLM Prompt Tuning

TL;DR

PromptTuner is an SLO-aware elastic system that optimizes large language model prompt tuning by using a prompt bank and workload scheduler, significantly reducing violations and costs in prompt tuning services.

Contribution

It introduces a novel system with a prompt bank and workload scheduler to improve SLO adherence and cost efficiency in LLM prompt tuning workloads.

Findings

Reduces SLO violations by up to 4.0x

Lowers resource costs by up to 4.5x

Outperforms existing systems like INFless and ElasticFlow

Abstract

Prompt tuning has become a prominent strategy for enhancing the performance of Large Language Models (LLMs) on downstream tasks. Many IT enterprises now offer Prompt-Tuning-as-a-Service to fulfill the growing demand for prompt tuning LLMs on downstream tasks. Their primary objective is to satisfy users Service Level Objectives (SLOs) while reducing resource provisioning costs. Nevertheless, our characterization analysis for existing deep learning resource management systems reveals that they are insufficient to optimize these objectives for LLM prompt tuning workloads. In this paper, we introduce PromptTuner, an SLO-aware elastic system to optimize LLM prompt tuning. It contains two innovations. (1) We design a Prompt Bank to identify efficient initial prompts to expedite the convergence of prompt tuning. (2) We develop aWorkload Scheduler to enable fast resource allocation to reduce the SLO violation and resource costs. In our evaluation, PromptTuner reduces SLO violations by 4.0x and 7.9x, and lowers costs by 1.6x and 4.5x, compared to INFless and ElasticFlow respectively.