Charged Point Normalization: An Efficient Solution to the Saddle Point Problem

TL;DR

This paper proposes Charged Point Normalization, a novel dynamic normalization method that helps neural networks escape saddle points without using second-order information, significantly improving training efficiency across various datasets.

Contribution

It introduces Charged Point Normalization, a new normalization technique that enhances saddle point escape in neural network training without relying on second-order derivatives.

Findings

Drastically improves training performance on deep neural networks.

Effective across various datasets and network architectures.

Eliminates the need for second-order information in saddle point escape.

Abstract

Recently, the problem of local minima in very high dimensional non-convex optimization has been challenged and the problem of saddle points has been introduced. This paper introduces a dynamic type of normalization that forces the system to escape saddle points. Unlike other saddle point escaping algorithms, second order information is not utilized, and the system can be trained with an arbitrary gradient descent learner. The system drastically improves learning in a range of deep neural networks on various data-sets in comparison to non-CPN neural networks.