DDS-NAS: Dynamic Data Selection within Neural Architecture Search via On-line Hard Example Mining applied to Image Classification

TL;DR

DDS-NAS introduces a dynamic data selection method using on-line hard example mining and curriculum learning to significantly accelerate neural architecture search for image classification without sacrificing accuracy.

Contribution

It presents a novel framework combining autoencoder-based similarity embedding, kd-tree data structure, and curriculum learning to speed up NAS training.

Findings

Speeds up NAS training by up to 27x

Reduces training duration and iterations for convergence

Maintains performance while improving efficiency

Abstract

In order to address the scalability challenge within Neural Architecture Search (NAS), we speed up NAS training via dynamic hard example mining within a curriculum learning framework. By utilizing an autoencoder that enforces an image similarity embedding in latent space, we construct an efficient kd-tree structure to order images by furthest neighbour dissimilarity in a low-dimensional embedding. From a given query image from our subsample dataset, we can identify the most dissimilar image within the global dataset in logarithmic time. Via curriculum learning, we then dynamically re-formulate an unbiased subsample dataset for NAS optimisation, upon which the current NAS solution architecture performs poorly. We show that our DDS-NAS framework speeds up gradient-based NAS strategies by up to 27x without loss in performance. By maximising the contribution of each image sample during training, we reduce the duration of a NAS training cycle and the number of iterations required for convergence.