Scalable AI Inference: Performance Analysis and Optimization of AI Model Serving

TL;DR

This paper evaluates and optimizes a BentoML-based AI inference system for scalable model serving, analyzing performance under realistic workloads and proposing improvements for efficiency and resilience.

Contribution

It provides a comprehensive performance analysis and optimization strategies for scalable AI inference using BentoML in real-world scenarios.

Findings

Optimizations significantly reduce latency and increase throughput.

Performance varies with workload intensity and distribution.

Deployment in a K3s cluster enhances system resilience.

Abstract

AI research often emphasizes model design and algorithmic performance, while deployment and inference remain comparatively underexplored despite being critical for real-world use. This study addresses that gap by investigating the performance and optimization of a BentoML-based AI inference system for scalable model serving developed in collaboration with graphworks.ai. The evaluation first establishes baseline performance under three realistic workload scenarios. To ensure a fair and reproducible assessment, a pre-trained RoBERTa sentiment analysis model is used throughout the experiments. The system is subjected to traffic patterns following gamma and exponential distributions in order to emulate real-world usage conditions, including steady, bursty, and high-intensity workloads. Key performance metrics, such as latency percentiles and throughput, are collected and analyzed to identify bottlenecks in the inference pipeline. Based on the baseline results, optimization strategies are introduced at multiple levels of the serving stack to improve efficiency and scalability. The optimized system is then reevaluated under the same workload conditions, and the results are compared with the baseline using statistical analysis to quantify the impact of the applied improvements. The findings demonstrate practical strategies for achieving efficient and scalable AI inference with BentoML. The study examines how latency and throughput scale under varying workloads, how optimizations at the runtime, service, and deployment levels affect response time, and how deployment in a single-node K3s cluster influences resilience during disruptions.