AGBoost: Attention-based Modification of Gradient Boosting Machine

TL;DR

AGBoost introduces an attention mechanism into gradient boosting machines, assigning trainable weights to iterations, which improves regression performance and can be efficiently optimized using quadratic programming.

Contribution

The paper presents a novel attention-based modification to GBM that incorporates trainable weights for iterations, enhancing model flexibility and performance.

Findings

Attention weights depend linearly on trainable parameters.

The model is optimized via quadratic programming with linear constraints.

Numerical experiments show improved regression accuracy.

Abstract

A new attention-based model for the gradient boosting machine (GBM) called AGBoost (the attention-based gradient boosting) is proposed for solving regression problems. The main idea behind the proposed AGBoost model is to assign attention weights with trainable parameters to iterations of GBM under condition that decision trees are base learners in GBM. Attention weights are determined by applying properties of decision trees and by using the Huber's contamination model which provides an interesting linear dependence between trainable parameters of the attention and the attention weights. This peculiarity allows us to train the attention weights by solving the standard quadratic optimization problem with linear constraints. The attention weights also depend on the discount factor as a tuning parameter, which determines how much the impact of the weight is decreased with the number of iterations. Numerical experiments performed for two types of base learners, original decision trees and extremely randomized trees with various regression datasets illustrate the proposed model.