SAMEdge: An Edge-cloud Video Analytics Architecture for the Segment Anything Model

TL;DR

SAMEdge is an innovative edge-cloud architecture that enhances real-time, prompt-based video analytics using the Segment Anything Model, balancing accuracy and latency in resource-constrained environments.

Contribution

It introduces new modules and algorithms for prompt and image encoding to optimize SAM computations on edge-cloud systems, enabling practical real-time applications.

Findings

Significantly improves analytics accuracy across different network bandwidths.

Effectively manages resource challenges in prompt and image encoding.

Demonstrates practical application with a Visual Tour Guide case study.

Abstract

As artificial intelligence continues to evolve, it is increasingly capable of handling a wide range of video analytics tasks with merely one large model. One of the key foundation technologies is the Segment Anything Model (SAM), which allows the video analytics tasks to be determined on the fly according to the input prompts from the user. However, achieving real-time response in video analytics applications is crucial for user experiences due to the limited communication and computation resources on the edge, especially with SAM, where users may continuously interact by adding or adjusting prompts. In this paper, we propose SAMEdge, a novel edge-cloud computing architecture designed to support SAM computations for edge users. SAMEdge integrates new modules on the edge and the cloud to maximize analytics accuracy under visual prompts and image prompts input with latency constraints. It addresses resource challenges associated with prompt encoding and image encoding by offering a visual prompt transformation algorithm for visual prompts and efficient workload partitioning for image encoding. SAMEdge is implemented by extending the open-source SAM project from Meta AI. We demonstrate the practical application of SAMEdge through a case study on a Visual Tour Guide application. Our evaluation indicates that SAMEdge significantly enhances the accuracy of the video analytics application under distinct network bandwidths across various prompts.