Probabilistic Pretraining for Neural Regression

TL;DR

This paper introduces NIAQUE, a novel neural model for probabilistic regression that leverages transfer learning through pre-training and fine-tuning, demonstrating improved performance across diverse datasets and competitive Kaggle results.

Contribution

The work presents NIAQUE, a new permutation-invariant neural model for probabilistic regression that enables effective transfer learning via pre-training and fine-tuning.

Findings

Pre-training NIAQUE improves regression performance on various datasets.

NIAQUE outperforms strong baselines in Kaggle competitions.

Transfer learning enhances probabilistic regression accuracy.

Abstract

Transfer learning for probabilistic regression remains underexplored. This work closes this gap by introducing NIAQUE, Neural Interpretable Any-Quantile Estimation, a new model designed for transfer learning in probabilistic regression through permutation invariance. We demonstrate that pre-training NIAQUE directly on diverse downstream regression datasets and fine-tuning it on a specific target dataset enhances performance on individual regression tasks, showcasing the positive impact of probabilistic transfer learning. Furthermore, we highlight the effectiveness of NIAQUE in Kaggle competitions against strong baselines involving tree-based models and recent neural foundation models TabPFN and TabDPT. The findings highlight NIAQUE's efficacy as a robust and scalable framework for probabilistic regression, leveraging transfer learning to enhance predictive performance.