Approximated Orthogonal Projection Unit: Stabilizing Regression Network Training Using Natural Gradient

TL;DR

This paper introduces the Approximated Orthogonal Projection Unit (AOPU), a neural network component that improves training stability and robustness by approximating natural gradient updates, especially beneficial for industrial soft sensor applications.

Contribution

The paper presents AOPU, a novel neural network module with a solid mathematical foundation that enhances training stability and approximates natural gradient descent.

Findings

AOPU achieves more stable convergence in training.

AOPU outperforms existing models on chemical process datasets.

AOPU provides a robust training mechanism for soft sensor networks.

Abstract

Neural networks (NN) are extensively studied in cutting-edge soft sensor models due to their feature extraction and function approximation capabilities. Current research into network-based methods primarily focuses on models' offline accuracy. Notably, in industrial soft sensor context, online optimizing stability and interpretability are prioritized, followed by accuracy. This requires a clearer understanding of network's training process. To bridge this gap, we propose a novel NN named the Approximated Orthogonal Projection Unit (AOPU) which has solid mathematical basis and presents superior training stability. AOPU truncates the gradient backpropagation at dual parameters, optimizes the trackable parameters updates, and enhances the robustness of training. We further prove that AOPU attains minimum variance estimation (MVE) in NN, wherein the truncated gradient approximates the natural gradient (NG). Empirical results on two chemical process datasets clearly show that AOPU outperforms other models in achieving stable convergence, marking a significant advancement in soft sensor field.