Scalable Engine and the Performance of Different LLM Models in a SLURM based HPC architecture

TL;DR

This paper presents a scalable HPC-based architecture using SLURM for deploying and managing heterogeneous LLMs efficiently, demonstrating high throughput and low latency for small models and identifying saturation points for larger models.

Contribution

It introduces a novel scalable inference engine architecture leveraging SLURM, containerization, and REST APIs for efficient deployment and management of diverse LLMs in HPC environments.

Findings

Small models handle up to 128 concurrent requests with sub-50 ms latency.

Larger models saturate with as few as two concurrent users, with over 2 seconds latency.

The architecture scales reliably for batch and interactive inference scenarios.

Abstract

This work elaborates on a High performance computing (HPC) architecture based on Simple Linux Utility for Resource Management (SLURM) [1] for deploying heterogeneous Large Language Models (LLMs) into a scalable inference engine. Dynamic resource scheduling and seamless integration of containerized microservices have been leveraged herein to manage CPU, GPU, and memory allocations efficiently in multi-node clusters. Extensive experiments, using Llama 3.2 (1B and 3B parameters) [2] and Llama 3.1 (8B and 70B) [3], probe throughput, latency, and concurrency and show that small models can handle up to 128 concurrent requests at sub-50 ms latency, while for larger models, saturation happens with as few as two concurrent users, with a latency of more than 2 seconds. This architecture includes Representational State Transfer Application Programming Interfaces (REST APIs) [4] endpoints for single and bulk inferences, as well as advanced workflows such as multi-step "tribunal" refinement. Experimental results confirm minimal overhead from container and scheduling activities and show that the approach scales reliably both for batch and interactive settings. We further illustrate real-world scenarios, including the deployment of chatbots with retrievalaugmented generation, which helps to demonstrate the flexibility and robustness of the architecture. The obtained results pave ways for significantly more efficient, responsive, and fault-tolerant LLM inference on large-scale HPC infrastructures.