NNStreamer: Efficient and Agile Development of On-Device AI Systems

TL;DR

NNStreamer is an open-source framework that streamlines on-device AI development by integrating neural networks into stream pipelines, improving performance, reducing costs, and simplifying implementation across diverse devices.

Contribution

It introduces a novel stream processing approach for neural networks, enabling efficient, flexible, and reusable on-device AI system development.

Findings

Significant performance improvements in on-device neural network processing

Reduction in development costs for AI applications

Successful deployment across multiple consumer devices

Abstract

We propose NNStreamer, a software system that handles neural networks as filters of stream pipelines, applying the stream processing paradigm to deep neural network applications. A new trend with the wide-spread of deep neural network applications is on-device AI. It is to process neural networks on mobile devices or edge/IoT devices instead of cloud servers. Emerging privacy issues, data transmission costs, and operational costs signify the need for on-device AI, especially if we deploy a massive number of devices. NNStreamer efficiently handles neural networks with complex data stream pipelines on devices, significantly improving the overall performance with minimal efforts. Besides, NNStreamer simplifies implementations and allows reusing off-the-shelf media filters directly, which reduces developmental costs significantly. We are already deploying NNStreamer for a wide range of products and platforms, including the Galaxy series and various consumer electronic devices. The experimental results suggest a reduction in developmental costs and enhanced performance of pipeline architectures and NNStreamer. It is an open-source project incubated by Linux Foundation AI, available to the public and applicable to various hardware and software platforms.