ORXE: Orchestrating Experts for Dynamically Configurable Efficiency

TL;DR

ORXE is a flexible, training-free framework that dynamically orchestrates pre-trained experts to optimize inference efficiency and accuracy in real-time, adaptable to diverse input complexities and resource constraints.

Contribution

This paper introduces ORXE, a novel, training-free system that dynamically manages multiple experts for efficient AI inference without complex metamodels.

Findings

ORXE outperforms individual experts in efficiency and accuracy.

It adapts to various device capabilities and input complexities.

The system maintains high performance with adjustable cost-performance trade-offs.

Abstract

This paper presents ORXE, a modular and adaptable framework for achieving real-time configurable efficiency in AI models. By leveraging a collection of pre-trained experts with diverse computational costs and performance levels, ORXE dynamically adjusts inference pathways based on the complexity of input samples. Unlike conventional approaches that require complex metamodel training, ORXE achieves high efficiency and flexibility without complicating the development process. The proposed system utilizes a confidence-based gating mechanism to allocate appropriate computational resources for each input. ORXE also supports adjustments to the preference between inference cost and prediction performance across a wide range during runtime. We implemented a training-free ORXE system for image classification tasks, evaluating its efficiency and accuracy across various devices. The results demonstrate that ORXE achieves superior performance compared to individual experts and other dynamic models in most cases. This approach can be extended to other applications, providing a scalable solution for diverse real-world deployment scenarios.