Toward Among-Device AI from On-Device AI with Stream Pipelines

TL;DR

This paper proposes extensions to the NNStreamer framework to enable among-device AI, allowing AI pipelines to share resources across diverse devices, enhancing flexibility and performance of on-device AI systems.

Contribution

The paper introduces novel extensions to NNStreamer that facilitate among-device AI, supporting cross-device resource sharing and interoperability in on-device AI applications.

Findings

Extended NNStreamer enables cross-device AI pipeline connectivity

Improved resource sharing across heterogeneous devices

Open source contribution to the Linux Foundation

Abstract

Modern consumer electronic devices often provide intelligence services with deep neural networks. We have started migrating the computing locations of intelligence services from cloud servers (traditional AI systems) to the corresponding devices (on-device AI systems). On-device AI systems generally have the advantages of preserving privacy, removing network latency, and saving cloud costs. With the emergent of on-device AI systems having relatively low computing power, the inconsistent and varying hardware resources and capabilities pose difficulties. Authors' affiliation has started applying a stream pipeline framework, NNStreamer, for on-device AI systems, saving developmental costs and hardware resources and improving performance. We want to expand the types of devices and applications with on-device AI services products of both the affiliation and second/third parties. We also want to make each AI service atomic, re-deployable, and shared among connected devices of arbitrary vendors; we now have yet another requirement introduced as it always has been. The new requirement of "among-device AI" includes connectivity between AI pipelines so that they may share computing resources and hardware capabilities across a wide range of devices regardless of vendors and manufacturers. We propose extensions of the stream pipeline framework, NNStreamer, for on-device AI so that NNStreamer may provide among-device AI capability. This work is a Linux Foundation (LF AI and Data) open source project accepting contributions from the general public.