Orpheus: A New Deep Learning Framework for Easy Deployment and Evaluation of Edge Inference

TL;DR

Orpheus is a lightweight deep learning framework designed to simplify deployment and evaluation of inference optimizations on edge devices, addressing compatibility and complexity issues present in existing systems.

Contribution

It introduces a minimalistic framework with easy integration capabilities, facilitating research and deployment of edge inference optimizations.

Findings

Preliminary evaluation shows promising performance.

Framework simplifies deployment on constrained devices.

Supports easy integration with third-party systems.

Abstract

Optimising deep learning inference across edge devices and optimisation targets such as inference time, memory footprint and power consumption is a key challenge due to the ubiquity of neural networks. Today, production deep learning frameworks provide useful abstractions to aid machine learning engineers and systems researchers. However, in exchange they can suffer from compatibility challenges (especially on constrained platforms), inaccessible code complexity, or design choices that otherwise limit research from a systems perspective. This paper presents Orpheus, a new deep learning framework for easy prototyping, deployment and evaluation of inference optimisations. Orpheus features a small codebase, minimal dependencies, and a simple process for integrating other third party systems. We present some preliminary evaluation results.