Neural Architecture Search with Mixed Bio-inspired Learning Rules

TL;DR

This paper introduces a neural architecture search method that automatically combines different bio-inspired learning rules across layers, resulting in models that outperform single-rule bio-inspired networks and sometimes surpass back-propagation models in accuracy.

Contribution

The paper presents a novel NAS approach that discovers optimal layer-wise combinations of bio-inspired learning rules, improving accuracy and scalability of bio-inspired neural networks.

Findings

Achieved new record accuracies for bio-inspired models on CIFAR-10, CIFAR-100, ImageNet16-120, and ImageNet.

Layer-wise diversity in learning rules enhances model performance and robustness.

Some mixed-rule models outperform comparable back-propagation networks.

Abstract

Bio-inspired neural networks are attractive for their adversarial robustness, energy frugality, and closer alignment with cortical physiology, yet they often lag behind back-propagation (BP) based models in accuracy and ability to scale. We show that allowing the use of different bio-inspired learning rules in different layers, discovered automatically by a tailored neural-architecture-search (NAS) procedure, bridges this gap. Starting from standard NAS baselines, we enlarge the search space to include bio-inspired learning rules and use NAS to find the best architecture and learning rule to use in each layer. We show that neural networks that use different bio-inspired learning rules for different layers have better accuracy than those that use a single rule across all the layers. The resulting NN that uses a mix of bio-inspired learning rules sets new records for bio-inspired models: 95.16% on CIFAR-10, 76.48% on CIFAR-100, 43.42% on ImageNet16-120, and 60.51% top-1 on ImageNet. In some regimes, they even surpass comparable BP-based networks while retaining their robustness advantages. Our results suggest that layer-wise diversity in learning rules allows better scalability and accuracy, and motivates further research on mixing multiple bio-inspired learning rules in the same network.