SuperSONIC: Cloud-Native Infrastructure for ML Inferencing

TL;DR

SuperSONIC is a scalable, cloud-native infrastructure that enhances ML inference deployment efficiency across scientific experiments by leveraging coprocessors, Kubernetes, and NVIDIA Triton, enabling flexible, high-throughput processing.

Contribution

It introduces SuperSONIC, a scalable server infrastructure that standardizes and optimizes ML inference deployment on cloud-native platforms for scientific research.

Findings

Successfully deployed for CERN experiments and astrophysics observatories.

Enhanced resource utilization and throughput in ML inference workflows.

Demonstrated scalability and flexibility across diverse scientific domains.

Abstract

The increasing computational demand from growing data rates and complex machine learning (ML) algorithms in large-scale scientific experiments has driven the adoption of the Services for Optimized Network Inference on Coprocessors (SONIC) approach. SONIC accelerates ML inference by offloading it to local or remote coprocessors to optimize resource utilization. Leveraging its portability to different types of coprocessors, SONIC enhances data processing and model deployment efficiency for cutting-edge research in high energy physics (HEP) and multi-messenger astrophysics (MMA). We developed the SuperSONIC project, a scalable server infrastructure for SONIC, enabling the deployment of computationally intensive tasks to Kubernetes clusters equipped with graphics processing units (GPUs). Using NVIDIA Triton Inference Server, SuperSONIC decouples client workflows from server infrastructure, standardizing communication, optimizing throughput, load balancing, and monitoring. SuperSONIC has been successfully deployed for the CMS and ATLAS experiments at the CERN Large Hadron Collider (LHC), the IceCube Neutrino Observatory (IceCube), and the Laser Interferometer Gravitational-Wave Observatory (LIGO) and tested on Kubernetes clusters at Purdue University, the National Research Platform (NRP), and the University of Chicago. SuperSONIC addresses the challenges of the Cloud-native era by providing a reusable, configurable framework that enhances the efficiency of accelerator-based inference deployment across diverse scientific domains and industries.