Supervised Score-Based Modeling by Gradient Boosting

TL;DR

This paper introduces a Supervised Score-based Model (SSM) that combines score matching with gradient boosting, achieving improved prediction accuracy and reduced inference time compared to existing probabilistic models.

Contribution

The paper proposes a novel SSM that integrates score matching with gradient boosting, providing a theoretical analysis and demonstrating superior performance over existing models.

Findings

Outperforms NGboost, CARD, and DBT in accuracy.

Reduces inference time compared to stochastic models.

Shows strong results in ablation experiments.

Abstract

Score-based generative models can effectively learn the distribution of data by estimating the gradient of the distribution. Due to the multi-step denoising characteristic, researchers have recently considered combining score-based generative models with the gradient boosting algorithm, a multi-step supervised learning algorithm, to solve supervised learning tasks. However, existing generative model algorithms are often limited by the stochastic nature of the models and the long inference time, impacting prediction performances. Therefore, we propose a Supervised Score-based Model (SSM), which can be viewed as a gradient boosting algorithm combining score matching. We provide a theoretical analysis of learning and sampling for SSM to balance inference time and prediction accuracy. Via the ablation experiment in selected examples, we demonstrate the outstanding performances of the proposed techniques. Additionally, we compare our model with other probabilistic models, including Natural Gradient Boosting (NGboost), Classification and Regression Diffusion Models (CARD), Diffusion Boosted Trees (DBT), and non-probabilistic GBM models. The experimental results show that our model outperforms existing models in both accuracy and inference time.