An Initialization Schema for Neuronal Networks on Tabular Data

TL;DR

This paper introduces a binomial initialization schema for neural networks tailored to tabular data, demonstrating improved performance over existing neural methods and enabling ensemble training with gradient masking.

Contribution

The paper proposes a novel binomial initialization approach for neural networks on tabular data and extends it for ensemble training with gradient masking.

Findings

Improved neural network performance on tabular datasets.

Effective ensemble training using the proposed initialization and gradient masking.

Discussion of limitations and future research directions.

Abstract

Nowadays, many modern applications require heterogeneous tabular data, which is still a challenging task in terms of regression and classification. Many approaches have been proposed to adapt neural networks for this task, but still, boosting and bagging of decision trees are the best-performing methods for this task. In this paper, we show that a binomial initialized neural network can be used effectively on tabular data. The proposed approach shows a simple but effective approach for initializing the first hidden layer in neural networks. We also show that this initializing schema can be used to jointly train ensembles by adding gradient masking to batch entries and using the binomial initialization for the last layer in a neural network. For this purpose, we modified the hinge binary loss and the soft max loss to make them applicable for joint ensemble training. We evaluate our approach on multiple public datasets and showcase the improved performance compared to other neural network-based approaches. In addition, we discuss the limitations and possible further research of our approach for improving the applicability of neural networks to tabular data. Link: https://es-cloud.cs.uni-tuebingen.de/d/8e2ab8c3fdd444e1a135/?p=%2FInitializationNeuronalNetworksTabularData&mode=list