On the Bounds of Function Approximations

TL;DR

This paper establishes a formal framework to analyze the computational limits of Neural Architecture Search (NAS) by reformulating it as a function approximation problem and exploring conditions for efficient solutions.

Contribution

It introduces a formalized framework for understanding NAS bounds, reformulates NAS as a function approximation problem, and identifies conditions for polynomial-time solutions.

Findings

FA is computationally infeasible to solve exactly for all functions.

Error minimization is possible if specific functions are in the search space.

Conditions are identified under which NAS can be solved in polynomial time.

Abstract

Within machine learning, the subfield of Neural Architecture Search (NAS) has recently garnered research attention due to its ability to improve upon human-designed models. However, the computational requirements for finding an exact solution to this problem are often intractable, and the design of the search space still requires manual intervention. In this paper we attempt to establish a formalized framework from which we can better understand the computational bounds of NAS in relation to its search space. For this, we first reformulate the function approximation problem in terms of sequences of functions, and we call it the Function Approximation (FA) problem; then we show that it is computationally infeasible to devise a procedure that solves FA for all functions to zero error, regardless of the search space. We show also that such error will be minimal if a specific class of functions is present in the search space. Subsequently, we show that machine learning as a mathematical problem is a solution strategy for FA, albeit not an effective one, and further describe a stronger version of this approach: the Approximate Architectural Search Problem (a-ASP), which is the mathematical equivalent of NAS. We leverage the framework from this paper and results from the literature to describe the conditions under which a-ASP can potentially solve FA as well as an exhaustive search, but in polynomial time.