An Approximation Algorithm for Optimal Subarchitecture Extraction

TL;DR

This paper introduces an approximation algorithm for selecting optimal neural network architectures based on size, speed, and accuracy, providing near-optimal solutions efficiently for many instances.

Contribution

It presents a novel approximation algorithm that behaves like an FPTAS for a broad class of instances in neural architecture optimization.

Findings

Algorithm achieves approximation error   |1 - | for many instances.

Runs in polynomial time with respect to input parameters.

Provides a formal framework for optimal subarchitecture extraction.

Abstract

We consider the problem of finding the set of architectural parameters for a chosen deep neural network which is optimal under three metrics: parameter size, inference speed, and error rate. In this paper we state the problem formally, and present an approximation algorithm that, for a large subset of instances behaves like an FPTAS with an approximation error of $\rho \leq |{1- \epsilon}|$, and that runs in $O(|{\Xi}| + |{W^*_T}|(1 + |{\Theta}||{B}||{\Xi}|/({\epsilon\, s^{3/2})}))$ steps, where $\epsilon$ and $s$ are input parameters; $|{B}|$ is the batch size; $|{W^*_T}|$ denotes the cardinality of the largest weight set assignment; and $|{\Xi}|$ and $|{\Theta}|$ are the cardinalities of the candidate architecture and hyperparameter spaces, respectively.