Evolving Neural Selection with Adaptive Regularization

TL;DR

This paper introduces Adaptive Neural Selection (ANS), a framework that dynamically adjusts neuron importance in deep networks based on input difficulty, improving performance on image recognition tasks.

Contribution

The paper proposes a novel ANS framework that evolves neuron selection to adapt to input complexity, enhancing neural network effectiveness.

Findings

Significant performance improvements on image recognition benchmarks.

Validation of each component's contribution through ablation studies.

Demonstrated adaptability of neuron selection to input difficulty.

Abstract

Over-parameterization is one of the inherent characteristics of modern deep neural networks, which can often be overcome by leveraging regularization methods, such as Dropout. Usually, these methods are applied globally and all the input cases are treated equally. However, given the natural variation of the input space for real-world tasks such as image recognition and natural language understanding, it is unlikely that a fixed regularization pattern will have the same effectiveness for all the input cases. In this work, we demonstrate a method in which the selection of neurons in deep neural networks evolves, adapting to the difficulty of prediction. We propose the Adaptive Neural Selection (ANS) framework, which evolves to weigh neurons in a layer to form network variants that are suitable to handle different input cases. Experimental results show that the proposed method can significantly improve the performance of commonly-used neural network architectures on standard image recognition benchmarks. Ablation studies also validate the effectiveness and contribution of each component in the proposed framework.