Hera: A Heterogeneity-Aware Multi-Tenant Inference Server for Personalized Recommendations

TL;DR

Hera is a heterogeneity-aware multi-tenant inference server that optimizes resource allocation for recommendation models, significantly improving server utilization and reducing costs while maintaining low latency.

Contribution

Hera introduces a heterogeneity-aware approach to co-locate recommendation models, balancing latency and throughput for multi-tenant inference.

Findings

37.3% improvement in machine utilization

26% reduction in server count

Significant latency and throughput improvements

Abstract

While providing low latency is a fundamental requirement in deploying recommendation services, achieving high resource utility is also crucial in cost-effectively maintaining the datacenter. Co-locating multiple workers of a model is an effective way to maximize query-level parallelism and server throughput, but the interference caused by concurrent workers at shared resources can prevent server queries from meeting its SLA. Hera utilizes the heterogeneous memory requirement of multi-tenant recommendation models to intelligently determine a productive set of co-located models and its resource allocation, providing fast response time while achieving high throughput. We show that Hera achieves an average 37.3% improvement in effective machine utilization, enabling 26% reduction in required servers, significantly improving upon the baseline recommedation inference server.