Neural Architecture Search of Deep Priors: Towards Continual Learning without Catastrophic Interference

TL;DR

This paper demonstrates that neural architecture search can identify deep priors that enable effective classification without training, and can facilitate continual learning without catastrophic interference.

Contribution

It introduces a method to find deep priors via neural architecture search that perform comparably to trained networks without inference, and extends this to incremental learning without catastrophic interference.

Findings

Random weight architectures can achieve competitive classification performance.

Deep priors do not require additional inference or training.

Deep priors enable incremental learning without catastrophic interference.

Abstract

In this paper we analyze the classification performance of neural network structures without parametric inference. Making use of neural architecture search, we empirically demonstrate that it is possible to find random weight architectures, a deep prior, that enables a linear classification to perform on par with fully trained deep counterparts. Through ablation experiments, we exclude the possibility of winning a weight initialization lottery and confirm that suitable deep priors do not require additional inference. In an extension to continual learning, we investigate the possibility of catastrophic interference free incremental learning. Under the assumption of classes originating from the same data distribution, a deep prior found on only a subset of classes is shown to allow discrimination of further classes through training of a simple linear classifier.