Self-Adaptive, Dynamic, Integrated Statistical and Information Theory Learning

TL;DR

This paper introduces a self-adaptive neural network training algorithm that dynamically integrates statistical and information theory measures, enhancing training efficiency and model accuracy.

Contribution

It proposes a novel error measure, $E_{ExpAbs}$, and incorporates it into a dynamic Levenberg-Marquardt algorithm, unifying statistical and information-theoretic approaches.

Findings

The new algorithm improves training process dynamics.

It achieves better model accuracy in tests.

It successfully combines statistical and information theory measures.

Abstract

The paper analyses and serves with a positioning of various error measures applied in neural network training and identifies that there is no best of measure, although there is a set of measures with changing superiorities in different learning situations. An outstanding, remarkable measure called $E_{Exp}$ published by Silva and his research partners represents a research direction to combine more measures successfully with fixed importance weighting during learning. The main idea of the paper is to go far beyond and to integrate this relative importance into the neural network training algorithm(s) realized through a novel error measure called $E_{ExpAbs}$. This approach is included into the Levenberg-Marquardt training algorithm, so, a novel version of it is also introduced, resulting a self-adaptive, dynamic learning algorithm. This dynamism does not has positive effects on the resulted model accuracy only, but also on the training process itself. The described comprehensive algorithm tests proved that the proposed, novel algorithm integrates dynamically the two big worlds of statistics and information theory that is the key novelty of the paper.