Hybrid Offline-online Scheduling Method for Large Language Model Inference Optimization

TL;DR

This paper introduces a hybrid offline-online scheduling approach for large language model inference that significantly enhances hardware utilization and reduces inference time through a mixed-integer programming model and dynamic scheduling techniques.

Contribution

It presents a novel hybrid offline-online scheduling method combining MIP modeling, theoretical lower bounds, and dynamic preemptive scheduling for LLM inference optimization.

Findings

System utilization improved from 80.2% to 89.1%.

Total inference time decreased from 201.00 to 190.58 seconds.

Method outperforms baseline in 100-case study with 8.0% average utilization increase.

Abstract

With the development of large language models (LLMs), it has become increasingly important to optimize hardware usage and improve throughput. In this paper, we study the inference optimization of the serving system that deploys LLMs. To optimize system throughput and maximize hardware utilization, we formulate the inference optimization problem as a mixed-integer programming (MIP) model and propose a hybrid offline-online method as solution. The offline method improves large-scale inference systems by introducing a Minimizing Makespan Bin Packing Problem. We further provide a theoretical lower bound computation method. Then, we propose an online sorting and preemptive scheduling method to better utilize hardware. In the online iteration scheduling process, a Lagrangian method is applied to evaluate the cost efficiency of inserting prefill stages versus decode stages at each iteration and dynamically determine when to preempt decoding tasks and insert prefill tasks. Experiments using real-world data from the LLaMA-65B model and the GSM8K dataset demonstrate that system utilization improves from 80.2% to 89.1%, and the total inference time decreases from 201.00 to 190.58 seconds. A 100-cases study shows that our method consistently outperforms the baseline method and improves the utilization rate by 8.0% on average. Finally, we discuss potential future extensions, including stochastic modeling, reinforcement learning-based schedulers, and dynamic decision-making strategies for system throughput and hardware utilization.