Designing a Linearized Potential Function in Neural Network Optimization Using Csisz\'{a}r Type of Tsallis Entropy

TL;DR

This paper introduces a novel framework using Csiszár type of Tsallis entropy to design a linearized potential function, leading to exponential convergence in neural network optimization.

Contribution

It proposes a new approach employing generalized Tsallis entropy for potential functions, addressing technical challenges in convergence analysis.

Findings

Established a framework with Tsallis entropy for potential functions.

Derived exponential convergence results for neural network optimization.

Enhanced understanding of entropy's role in convergence behavior.

Abstract

In recent years, learning for neural networks can be viewed as optimization in the space of probability measures. To obtain the exponential convergence to the optimizer, the regularizing term based on Shannon entropy plays an important role. Even though an entropy function heavily affects convergence results, there is almost no result on its generalization, because of the following two technical difficulties: one is the lack of sufficient condition for generalized logarithmic Sobolev inequality, and the other is the distributional dependence of the potential function within the gradient flow equation. In this paper, we establish a framework that utilizes a linearized potential function via Csisz\'{a}r type of Tsallis entropy, which is one of the generalized entropies. We also show that our new framework enable us to derive an exponential convergence result.