SensiX++: Bringing MLOPs and Multi-tenant Model Serving to Sensory Edge Devices

TL;DR

SensiX++ is a modular, multi-tenant runtime system that integrates MLOps for efficient, automated, and resource-aware model serving on edge sensory devices, reducing operational complexity.

Contribution

The paper introduces SensiX++, a novel system that combines modular componentisation, document-centric orchestration, and adaptive scheduling for multi-tenant model serving on edge devices.

Findings

Achieves high throughput across various models and configurations.

Reduces manual effort in deploying and managing models on edge devices.

Demonstrates significant efficiency improvements on Jetson AGX and Coral TPU.

Abstract

We present SensiX++ - a multi-tenant runtime for adaptive model execution with integrated MLOps on edge devices, e.g., a camera, a microphone, or IoT sensors. SensiX++ operates on two fundamental principles - highly modular componentisation to externalise data operations with clear abstractions and document-centric manifestation for system-wide orchestration. First, a data coordinator manages the lifecycle of sensors and serves models with correct data through automated transformations. Next, a resource-aware model server executes multiple models in isolation through model abstraction, pipeline automation and feature sharing. An adaptive scheduler then orchestrates the best-effort executions of multiple models across heterogeneous accelerators, balancing latency and throughput. Finally, microservices with REST APIs serve synthesised model predictions, system statistics, and continuous deployment. Collectively, these components enable SensiX++ to serve multiple models efficiently with fine-grained control on edge devices while minimising data operation redundancy, managing data and device heterogeneity, reducing resource contention and removing manual MLOps. We benchmark SensiX++ with ten different vision and acoustics models across various multi-tenant configurations on different edge accelerators (Jetson AGX and Coral TPU) designed for sensory devices. We report on the overall throughput and quantified benefits of various automation components of SensiX++ and demonstrate its efficacy to significantly reduce operational complexity and lower the effort to deploy, upgrade, reconfigure and serve embedded models on edge devices.