A Continuous Optimisation Benchmark Suite from Neural Network Regression

TL;DR

The paper introduces CORNN, a comprehensive benchmarking suite for evaluating continuous black-box optimization algorithms on neural network training problems, supporting diverse problem instances and architectures.

Contribution

It presents CORNN, a new benchmark suite for continuous optimization in neural network training, enabling systematic comparison of algorithms across varied problem settings.

Findings

Evolutionary algorithms show complementary strengths.

Gradient-based methods outperform some population-based algorithms.

CORNN facilitates standardized benchmarking for neural network optimization.

Abstract

Designing optimisation algorithms that perform well in general requires experimentation on a range of diverse problems. Training neural networks is an optimisation task that has gained prominence with the recent successes of deep learning. Although evolutionary algorithms have been used for training neural networks, gradient descent variants are by far the most common choice with their trusted good performance on large-scale machine learning tasks. With this paper we contribute CORNN (Continuous Optimisation of Regression tasks using Neural Networks), a large suite for benchmarking the performance of any continuous black-box algorithm on neural network training problems. Using a range of regression problems and neural network architectures, problem instances with different dimensions and levels of difficulty can be created. We demonstrate the use of the CORNN Suite by comparing the performance of three evolutionary and swarm-based algorithms on over 300 problem instances, showing evidence of performance complementarity between the algorithms. As a baseline, the performance of the best population-based algorithm is benchmarked against a gradient-based approach. The CORNN suite is shared as a public web repository to facilitate easy integration with existing benchmarking platforms.