JANUS: Benchmarking Commercial and Open-Source Cloud and Edge Platforms for Object and Anomaly Detection Workloads

TL;DR

This paper benchmarks various cloud and edge platforms for IoT workloads, comparing performance and costs of compute-light and compute-intensive tasks, including proprietary and open-source object detection solutions.

Contribution

It provides a comprehensive performance and cost analysis of multiple cloud and edge platforms for IoT workloads, highlighting the trade-offs between proprietary and open-source solutions.

Findings

AWS IoT Greengrass has at least 2X lower latency and 1.25X lower cost for outlier detection.

Open-source object detection on cloud VMs reduces costs but increases latency.

Edge deployment significantly reduces latency for video analytics.

Abstract

With diverse IoT workloads, placing compute and analytics close to where data is collected is becoming increasingly important. We seek to understand what is the performance and the cost implication of running analytics on IoT data at the various available platforms. These workloads can be compute-light, such as outlier detection on sensor data, or compute-intensive, such as object detection from video feeds obtained from drones. In our paper, JANUS, we profile the performance/$ and the compute versus communication cost for a compute-light IoT workload and a compute-intensive IoT workload. In addition, we also look at the pros and cons of some of the proprietary deep-learning object detection packages, such as Amazon Rekognition, Google Vision, and Azure Cognitive Services, to contrast with open-source and tunable solutions, such as Faster R-CNN (FRCNN). We find that AWS IoT Greengrass delivers at least 2X lower latency and 1.25X lower cost compared to all other cloud platforms for the compute-light outlier detection workload. For the compute-intensive streaming video analytics task, an opensource solution to object detection running on cloud VMs saves on dollar costs compared to proprietary solutions provided by Amazon, Microsoft, and Google, but loses out on latency (up to 6X). If it runs on a low-powered edge device, the latency is up to 49X lower.