Semi-adaptive Synergetic Two-way Pseudoinverse Learning System

TL;DR

This paper introduces a semi-adaptive, non-gradient learning system that simplifies hyperparameter tuning and enhances training efficiency through a data-driven, architecture design for deep learning models.

Contribution

It proposes a novel semi-adaptive two-way pseudoinverse learning system that eliminates the need for gradient descent, simplifying hyperparameter tuning and automating architecture design.

Findings

Outperforms traditional gradient-based methods in training efficiency.

Automates determination of subsystem depth using data-driven architecture.

Demonstrates effectiveness compared to mainstream non-gradient methods.

Abstract

Deep learning has become a crucial technology for making breakthroughs in many fields. Nevertheless, it still faces two important challenges in theoretical and applied aspects. The first lies in the shortcomings of gradient descent based learning schemes which are time-consuming and difficult to determine the learning control hyperparameters. Next, the architectural design of the model is usually tricky. In this paper, we propose a semi-adaptive synergetic two-way pseudoinverse learning system, wherein each subsystem encompasses forward learning, backward learning, and feature concatenation modules. The whole system is trained using a non-gradient descent learning algorithm. It simplifies the hyperparameter tuning while improving the training efficiency. The architecture of the subsystems is designed using a data-driven approach that enables automated determination of the depth of the subsystems. We compare our method with the baselines of mainstream non-gradient descent based methods and the results demonstrate the effectiveness of our proposed method. The source code for this paper is available at http://github.com/B-berrypie/Semi-adaptive-Synergetic-Two-way-Pseudoinverse-Learning-System}{http://github.com/B-berrypie/Semi-adaptive-Synergetic-Two-way-Pseudoinverse-Learning-System.