Sparse Mutation Decompositions: Fine Tuning Deep Neural Networks with Subspace Evolution

TL;DR

This paper introduces a method for fine-tuning deep neural networks using subspace evolution strategies that decompose mutations into low-dimensional components, improving efficiency and diversity.

Contribution

The paper presents a novel subspace decomposition approach for neuroevolutionary fine-tuning, enabling more efficient and targeted evolution of large pre-trained models.

Findings

Significant variance reduction in mutations

Improved fine-tuning performance on ImageNet

Enhanced ensemble diversity and accuracy

Abstract

Neuroevolution is a promising area of research that combines evolutionary algorithms with neural networks. A popular subclass of neuroevolutionary methods, called evolution strategies, relies on dense noise perturbations to mutate networks, which can be sample inefficient and challenging for large models with millions of parameters. We introduce an approach to alleviating this problem by decomposing dense mutations into low-dimensional subspaces. Restricting mutations in this way can significantly reduce variance as networks can handle stronger perturbations while maintaining performance, which enables a more controlled and targeted evolution of deep networks. This approach is uniquely effective for the task of fine tuning pre-trained models, which is an increasingly valuable area of research as networks continue to scale in size and open source models become more widely available. Furthermore, we show how this work naturally connects to ensemble learning where sparse mutations encourage diversity among children such that their combined predictions can reliably improve performance. We conduct the first large scale exploration of neuroevolutionary fine tuning and ensembling on the notoriously difficult ImageNet dataset, where we see small generalization improvements with only a single evolutionary generation using nearly a dozen different deep neural network architectures.