PERTINENCE: Input-based Opportunistic Neural Network Dynamic Execution

TL;DR

PERTINENCE is an online method that dynamically selects the most suitable neural network model based on input complexity, significantly reducing computational costs while maintaining accuracy across various CNNs and Vision Transformers.

Contribution

It introduces a genetic algorithm-based approach for input-aware model selection, optimizing the trade-off between accuracy and efficiency in neural network inference.

Findings

Achieves up to 36% reduction in operations with maintained accuracy.

Effective on CNNs trained on CIFAR-10/100 and Vision Transformers on TinyImageNet.

Provides alternative solutions to state-of-the-art models with improved efficiency.

Abstract

Deep neural networks (DNNs) have become ubiquitous thanks to their remarkable ability to model complex patterns across various domains such as computer vision, speech recognition, robotics, etc. While large DNN models are often more accurate than simpler, lightweight models, they are also resource- and energy-hungry. Hence, it is imperative to design methods to reduce reliance on such large models without significant degradation in output accuracy. The high computational cost of these models is often necessary only for a reduced set of challenging inputs, while lighter models can handle most simple ones. Thus, carefully combining properties of existing DNN models in a dynamic, input-based way opens opportunities to improve efficiency without impacting accuracy. In this work, we introduce PERTINENCE, a novel online method designed to analyze the complexity of input features and dynamically select the most suitable model from a pre-trained set to process a given input effectively. To achieve this, we employ a genetic algorithm to explore the training space of an ML-based input dispatcher, enabling convergence towards the Pareto front in the solution space that balances overall accuracy and computational efficiency. We showcase our approach on state-of-the-art Convolutional Neural Networks (CNNs) trained on the CIFAR-10 and CIFAR-100, as well as Vision Transformers (ViTs) trained on TinyImageNet dataset. We report results showing PERTINENCE's ability to provide alternative solutions to existing state-of-the-art models in terms of trade-offs between accuracy and number of operations. By opportunistically selecting among models trained for the same task, PERTINENCE achieves better or comparable accuracy with up to 36% fewer operations.