Dynamic Neural Network Architectural and Topological Adaptation and Related Methods -- A Survey

TL;DR

This survey reviews state-of-the-art techniques for adapting neural network architectures and topologies to reduce training and inference time and space, especially in resource-constrained environments.

Contribution

It provides a comprehensive overview and categorization of recent methods for dynamic neural network adaptation and related techniques.

Findings

Categorizes various architectural adaptation methods

Highlights techniques for resource-efficient DNN training and inference

Identifies gaps and future directions in adaptive neural network research

Abstract

Training and inference in deep neural networks (DNNs) has, due to a steady increase in architectural complexity and data set size, lead to the development of strategies for reducing time and space requirements of DNN training and inference, which is of particular importance in scenarios where training takes place in resource constrained computation environments or inference is part of a time critical application. In this survey, we aim to provide a general overview and categorization of state-of-the-art (SOTA) of techniques to reduced DNN training and inference time and space complexities with a particular focus on architectural adaptions.