Reducing Neural Network Parameter Initialization Into an SMT Problem

TL;DR

This paper introduces a novel method for initializing deep neural network parameters by formulating the problem as an SMT solver, leading to improved performance over traditional random or zero initialization methods.

Contribution

It presents the first approach to reduce deep neural network parameter initialization to an SMT problem, enabling better initial weights and improved network performance.

Findings

SMT-based initialization outperforms random initialization.

Deep networks with diverse activation functions benefit from the proposed method.

The approach is effective for deep neural networks, unlike previous small-scale studies.

Abstract

Training a neural network (NN) depends on multiple factors, including but not limited to the initial weights. In this paper, we focus on initializing deep NN parameters such that it performs better, comparing to random or zero initialization. We do this by reducing the process of initialization into an SMT solver. Previous works consider certain activation functions on small NNs, however the studied NN is a deep network with different activation functions. Our experiments show that the proposed approach for parameter initialization achieves better performance comparing to randomly initialized networks.