Orthogonal Stochastic Configuration Networks with Adaptive Construction Parameter for Data Analytics

TL;DR

This paper introduces Orthogonal Stochastic Configuration Networks (OSCN), which use Gram-Schmidt orthogonalization to reduce network complexity and improve generalization, with adaptive parameters and incremental updates demonstrated on various datasets.

Contribution

The paper proposes OSCN, a novel variant of SCNs that filters low-quality nodes using orthogonalization and adaptive parameter setting, enhancing efficiency and model compactness.

Findings

OSCN achieves better generalization with fewer parameters.

Experimental results show improved accuracy on multiple datasets.

The incremental scheme enhances computational efficiency.

Abstract

As a randomized learner model, SCNs are remarkable that the random weights and biases are assigned employing a supervisory mechanism to ensure universal approximation and fast learning. However, the randomness makes SCNs more likely to generate approximate linear correlative nodes that are redundant and low quality, thereby resulting in non-compact network structure. In the light of a fundamental principle in machine learning, that is, a model with fewer parameters holds improved generalization. This paper proposes orthogonal SCN, termed OSCN, to filtrate out the low-quality hidden nodes for network structure reduction by incorporating Gram-Schmidt orthogonalization technology. The universal approximation property of OSCN and an adaptive setting for the key construction parameters have been presented in details. In addition, an incremental updating scheme is developed to dynamically determine the output weights, contributing to improved computational efficiency. Finally, experimental results on two numerical examples and several real-world regression and classification datasets substantiate the effectiveness and feasibility of the proposed approach.