Continuous Ant-Based Neural Topology Search

TL;DR

This paper presents CANTS, a novel neural architecture search algorithm inspired by ant colony optimization, capable of designing neural networks of arbitrary size efficiently and effectively, demonstrated on power system time series prediction tasks.

Contribution

CANTS introduces a continuous search space and a scalable, distributed ant colony optimization approach for neural architecture search, enabling automatic design of neural networks without size constraints.

Findings

CANTS outperforms or matches state-of-the-art methods on power system prediction tasks.

It requires fewer hyperparameters, simplifying the NAS process.

The method scales well with high-performance computing resources.

Abstract

This work introduces a novel, nature-inspired neural architecture search (NAS) algorithm based on ant colony optimization, Continuous Ant-based Neural Topology Search (CANTS), which utilizes synthetic ants that move over a continuous search space based on the density and distribution of pheromones, is strongly inspired by how ants move in the real world. The paths taken by the ant agents through the search space are utilized to construct artificial neural networks (ANNs). This continuous search space allows CANTS to automate the design of ANNs of any size, removing a key limitation inherent to many current NAS algorithms that must operate within structures with a size predetermined by the user. CANTS employs a distributed asynchronous strategy which allows it to scale to large-scale high performance computing resources, works with a variety of recurrent memory cell structures, and makes use of a communal weight sharing strategy to reduce training time. The proposed procedure is evaluated on three real-world, time series prediction problems in the field of power systems and compared to two state-of-the-art algorithms. Results show that CANTS is able to provide improved or competitive results on all of these problems, while also being easier to use, requiring half the number of user-specified hyper-parameters.