More layers! End-to-end regression and uncertainty on tabular data with deep learning

TL;DR

This paper demonstrates that deep neural networks, when combined with ensemble methods and hierarchical multitask learning, can effectively perform regression and uncertainty estimation on tabular data, challenging the dominance of decision trees.

Contribution

It introduces an end-to-end deep learning algorithm for tabular data regression with uncertainty estimation, leveraging ensembles, Gaussian parameter estimation, and multitask learning.

Findings

Top-3 leaderboard positions in Yandex Shifts Weather challenge

Effective regression with uncertainty on tabular data using deep learning

Deep learning's potential surpasses traditional methods in structured data

Abstract

This paper attempts to analyze the effectiveness of deep learning for tabular data processing. It is believed that decision trees and their ensembles is the leading method in this domain, and deep neural networks must be content with computer vision and so on. But the deep neural network is a framework for building gradient-based hierarchical representations, and this key feature should be able to provide the best processing of generic structured (tabular) data, not just image matrices and audio spectrograms. This problem is considered through the prism of the Weather Prediction track in the Yandex Shifts challenge (in other words, the Yandex Shifts Weather task). This task is a variant of the classical tabular data regression problem. It is also connected with another important problem: generalization and uncertainty in machine learning. This paper proposes an end-to-end algorithm for solving the problem of regression with uncertainty on tabular data, which is based on the combination of four ideas: 1) deep ensemble of self-normalizing neural networks, 2) regression as parameter estimation of the Gaussian target error distribution, 3) hierarchical multitask learning, and 4) simple data preprocessing. Three modifications of the proposed algorithm form the top-3 leaderboard of the Yandex Shifts Weather challenge respectively. This paper considers that this success has occurred due to the fundamental properties of the deep learning algorithm, and tries to prove this.