A Dual Process Model for Optimizing Cross Entropy in Neural Networks

TL;DR

This paper proposes a dual process model for neural network training that optimizes cross-entropy by balancing divergence minimization and entropy maximization, introducing new theoretical insights and practical parameters.

Contribution

It introduces a novel dual process framework involving Kullback-Leibler divergence and Shannon entropy, deriving optimal training parameters and incorporating the golden ratio and complex numbers.

Findings

Derives optimal learning rate and momentum for backpropagation.

Establishes an equilibrium state balancing divergence and entropy.

Integrates the golden ratio and complex numbers into machine learning theory.

Abstract

Minimizing cross-entropy is a widely used method for training artificial neural networks. Many training procedures based on backpropagation use cross-entropy directly as their loss function. Instead, this theoretical essay investigates a dual process model with two processes, in which one process minimizes the Kullback-Leibler divergence while its dual counterpart minimizes the Shannon entropy. Postulating that learning consists of two dual processes complementing each other, the model defines an equilibrium state for both processes in which the loss function assumes its minimum. An advantage of the proposed model is that it allows deriving the optimal learning rate and momentum weight to update network weights for backpropagation. Furthermore, the model introduces the golden ratio and complex numbers as important new concepts in machine learning.