Bayesian Neural Tree Models for Nonparametric Regression

TL;DR

This paper introduces hybrid Bayesian neural tree models that combine decision trees and neural networks, offering flexible, well-generalizing nonparametric regression methods with fewer parameters and effective feature selection.

Contribution

The paper presents novel hybrid Bayesian neural tree models that integrate decision trees and neural networks, demonstrating their consistency and optimal parameter selection.

Findings

Models perform well with limited training data

Models effectively select features and predict

Models generalize better than traditional methods

Abstract

Frequentist and Bayesian methods differ in many aspects, but share some basic optimal properties. In real-life classification and regression problems, situations exist in which a model based on one of the methods is preferable based on some subjective criterion. Nonparametric classification and regression techniques, such as decision trees and neural networks, have frequentist (classification and regression trees (CART) and artificial neural networks) as well as Bayesian (Bayesian CART and Bayesian neural networks) approaches to learning from data. In this work, we present two hybrid models combining the Bayesian and frequentist versions of CART and neural networks, which we call the Bayesian neural tree (BNT) models. Both models exploit the architecture of decision trees and have lesser number of parameters to tune than advanced neural networks. Such models can simultaneously perform feature selection and prediction, are highly flexible, and generalize well in settings with a limited number of training observations. We study the consistency of the proposed models, and derive the optimal value of an important model parameter. We also provide illustrative examples using a wide variety of real-life regression data sets.