Deeper Insights into Learning Performance of Stochastic Configuration Networks

TL;DR

This paper analyzes the supervisory mechanism in Stochastic Configuration Networks (SCNs), identifies limitations in basis function selection, and proposes a new method with improved efficiency and scalability demonstrated through benchmark experiments.

Contribution

It introduces a novel supervisory mechanism and a recursive Moore-Penrose inverse method to enhance basis function selection in SCNs, improving learning performance and computational efficiency.

Findings

RMPI-SCN outperforms conventional SCN in learning capability

The new method reduces computational complexity

Enhanced scalability for large-scale data modeling

Abstract

Stochastic Configuration Networks (SCNs) are a class of randomized neural networks that integrate randomized algorithms within an incremental learning framework. A defining feature of SCNs is the supervisory mechanism, which adaptively adjusts the distribution to generate effective random basis functions, thereby enabling error-free learning. In this paper, we present a comprehensive analysis of the impact of the supervisory mechanism on the learning performance of SCNs. Our findings reveal that the current SCN framework evaluates the effectiveness of each random basis function in reducing residual errors using a lower bound on its error reduction potential, which constrains SCNs' overall learning efficiency. Specifically, SCNs may fail to consistently select the most effective random candidate as the new basis function during each training iteration. To overcome this problem, we propose a novel method for evaluating the hidden layer's output matrix, supported by a new supervisory mechanism that accurately assesses the error reduction potential of random basis functions without requiring the computation of the Moore-Penrose inverse of the output matrix. This approach enhances the selection of basis functions, reducing computational complexity and improving the overall scalability and learning capabilities of SCNs. We introduce a Recursive Moore-Penrose Inverse-SCN (RMPI-SCN) training scheme based on the new supervisory mechanism and demonstrate its effectiveness through simulations over some benchmark datasets. Experiments show that RMPI-SCN outperforms the conventional SCN in terms of learning capability, underscoring its potential to advance the SCN framework for large-scale data modeling applications.