TimeBill: Time-Budgeted Inference for Large Language Models

TL;DR

TimeBill introduces a framework for LLM inference that predicts execution time and adaptively manages cache eviction to meet strict time budgets, enhancing response reliability in time-critical applications.

Contribution

It presents a novel time-budgeted inference method with a response length predictor and execution time estimator for adaptive cache management in LLMs.

Findings

Improves task completion rate under strict time constraints

Maintains response quality despite varying time budgets

Outperforms fixed-cache eviction strategies in experiments

Abstract

Large Language Models (LLMs) are increasingly deployed in time-critical systems, such as robotics, autonomous driving, embodied intelligence, and industrial automation, where generating accurate responses within a given time budget is crucial for decision-making, control, or safety-critical tasks. However, the auto-regressive generation process of LLMs makes it challenging to model and estimate the end-to-end execution time. Furthermore, existing efficient inference methods based on a fixed key-value (KV) cache eviction ratio struggle to adapt to varying tasks with diverse time budgets, where an improper eviction ratio may lead to incomplete inference or a drop in response performance. In this paper, we propose TimeBill, a novel time-budgeted inference framework for LLMs that balances the inference efficiency and response performance. To be more specific, we propose a fine-grained response length predictor (RLP) and an execution time estimator (ETE) to accurately predict the end-to-end execution time of LLMs. Following this, we develop a time-budgeted efficient inference approach that adaptively adjusts the KV cache eviction ratio based on execution time prediction and the given time budget. Finally, through extensive experiments, we demonstrate the advantages of TimeBill in improving task completion rate and maintaining response performance under various overrun strategies.