Canonical dual solutions to nonconvex radial basis neural network optimization problem

TL;DR

This paper introduces a generalized canonical duality theory to effectively solve the highly non-convex optimization problems in training Radial Basis Function Neural Networks, enabling classification of solutions and addressing global optimality.

Contribution

It develops a sequential canonical dual transformation approach that reformulates RBFNN training as a dual problem without duality gap, allowing for better solution classification.

Findings

Global minimizer may not be the best RBFNN solution

Canonical dual theory can classify local and global extrema

Method applies to Gaussian RBFs and potentially other neural networks

Abstract

Radial Basis Functions Neural Networks (RBFNNs) are tools widely used in regression problems. One of their principal drawbacks is that the formulation corresponding to the training with the supervision of both the centers and the weights is a highly non-convex optimization problem, which leads to some fundamentally difficulties for traditional optimization theory and methods. This paper presents a generalized canonical duality theory for solving this challenging problem. We demonstrate that by sequential canonical dual transformations, the nonconvex optimization problem of the RBFNN can be reformulated as a canonical dual problem (without duality gap). Both global optimal solution and local extrema can be classified. Several applications to one of the most used Radial Basis Functions, the Gaussian function, are illustrated. Our results show that even for one-dimensional case, the global minimizer of the nonconvex problem may not be the best solution to the RBFNNs, and the canonical dual theory is a promising tool for solving general neural networks training problems.