Search Spaces for Neural Model Training

TL;DR

This paper investigates how expanding search spaces in neural model training can improve optimization, enable effective sparse models, and facilitate hardware-efficient structures, encouraging exploration beyond large neural models.

Contribution

It introduces the concept of search spaces in neural training, demonstrating how augmenting them can improve sparse model performance and hardware compatibility.

Findings

Augmented search spaces lead to competitive sparse models.

Sparse models trained with expanded search spaces are hardware-tolerant.

The approach opens new avenues for efficient neural training and inference.

Abstract

While larger neural models are pushing the boundaries of what deep learning can do, often more weights are needed to train models rather than to run inference for tasks. This paper seeks to understand this behavior using search spaces -- adding weights creates extra degrees of freedom that form new paths for optimization (or wider search spaces) rendering neural model training more effective. We then show how we can augment search spaces to train sparse models attaining competitive scores across dozens of deep learning workloads. They are also are tolerant of structures targeting current hardware, opening avenues for training and inference acceleration. Our work encourages research to explore beyond massive neural models being used today.