The M-factor: A Novel Metric for Evaluating Neural Architecture Search in Resource-Constrained Environments

TL;DR

This paper introduces the M-factor, a new metric combining accuracy and size to evaluate neural architectures, especially for resource-limited environments, and compares various NAS techniques using this metric.

Contribution

The paper proposes the M-factor metric for balanced evaluation of NAS architectures, addressing limitations of existing metrics focused solely on accuracy.

Findings

Policy-Based Reinforcement Learning achieved the highest M-factor score of 0.84.

Regularised Evolution optimized within 20 trials, showing efficiency.

Random search performed comparably to complex algorithms when evaluated with the M-factor.

Abstract

Neural Architecture Search (NAS) aims to automate the design of deep neural networks. However, existing NAS techniques often focus on maximising accuracy, neglecting model efficiency. This limitation restricts their use in resource-constrained environments like mobile devices and edge computing systems. Moreover, current evaluation metrics prioritise performance over efficiency, lacking a balanced approach for assessing architectures suitable for constrained scenarios. To address these challenges, this paper introduces the M-factor, a novel metric combining model accuracy and size. Four diverse NAS techniques are compared: Policy-Based Reinforcement Learning, Regularised Evolution, Tree-structured Parzen Estimator (TPE), and Multi-trial Random Search. These techniques represent different NAS paradigms, providing a comprehensive evaluation of the M-factor. The study analyses ResNet configurations on the CIFAR-10 dataset, with a search space of 19,683 configurations. Experiments reveal that Policy-Based Reinforcement Learning and Regularised Evolution achieved M-factor values of 0.84 and 0.82, respectively, while Multi-trial Random Search attained 0.75, and TPE reached 0.67. Policy-Based Reinforcement Learning exhibited performance changes after 39 trials, while Regularised Evolution optimised within 20 trials. The research investigates the optimisation dynamics and trade-offs between accuracy and model size for each strategy. Findings indicate that, in some cases, random search performed comparably to more complex algorithms when assessed using the M-factor. These results highlight how the M-factor addresses the limitations of existing metrics by guiding NAS towards balanced architectures, offering valuable insights for selecting strategies in scenarios requiring both performance and efficiency.