Attempted Blind Constrained Descent Experiments

TL;DR

This paper explores blind constrained descent methods for learning neural network weights, discussing layer-wise and filter-wise training approaches, and compares experimental results using open-source code implementations.

Contribution

It introduces experimental variations of blind descent techniques, including layer-by-layer and filter-by-filter training, with open-source code for reproducibility.

Findings

Layer-wise training shows promising results.

Filter-by-filter training offers alternative convergence behaviors.

Experimental results highlight the effectiveness of constrained descent methods.

Abstract

Blind Descent uses constrained but, guided approach to learn the weights. The probability density function is non-zero in the infinite space of the dimension (case in point: Gaussians and normal probability distribution functions). In Blind Descent paper, some of the implicit ideas involving layer by layer training and filter by filter training (with different batch sizes) were proposed as probable greedy solutions. The results of similar experiments are discussed. Octave (and proposed PyTorch variants) source code of the experiments of this paper can be found at https://github.com/PrasadNR/Attempted-Blind-Constrained-Descent-Experiments-ABCDE- . This is compared against the ABCDE derivatives of the original PyTorch source code of https://github.com/akshat57/Blind-Descent .