A Telescopic Binary Learning Machine for Training Neural Networks

TL;DR

This paper introduces a telescopic binary local search algorithm for training neural networks, which adaptively increases binary precision to improve search efficiency and solution quality across various complex tasks.

Contribution

The paper presents a novel multi-scale stochastic local search method with adaptive binary precision for neural network training, enhancing search speed and solution quality.

Findings

Effective in complex benchmark tasks

Improves generalization performance

Outperforms traditional training methods

Abstract

This paper proposes a new algorithm based on multi-scale stochastic local search with binary representation for training neural networks. In particular, we study the effects of neighborhood evaluation strategies, the effect of the number of bits per weight and that of the maximum weight range used for mapping binary strings to real values. Following this preliminary investigation, we propose a telescopic multi-scale version of local search where the number of bits is increased in an adaptive manner, leading to a faster search and to local minima of better quality. An analysis related to adapting the number of bits in a dynamic way is also presented. The control on the number of bits, which happens in a natural manner in the proposed method, is effective to increase the generalization performance. Benchmark tasks include a highly non-linear artificial problem, a control problem requiring either feed-forward or recurrent architectures for feedback control, and challenging real-world tasks in different application domains. The results demonstrate the effectiveness of the proposed method.