ITBoost: Information-Theoretic Trust for Robust Boosting

TL;DR

ITBoost introduces an information-theoretic approach to enhance the robustness of gradient boosting algorithms against noisy labels by evaluating residual evolution patterns.

Contribution

It proposes a novel residual trajectory complexity measure using MDL, improving robustness in noisy label scenarios compared to existing methods.

Findings

ITBoost outperforms leading boosting models on noisy tabular data.

It maintains high accuracy on clean data.

Provides a tighter generalization bound under label noise.

Abstract

Gradient boosting remains a strong and widely used method for tabular data learning, but its performance often degrades when training labels are noisy. This behavior is largely related to the way boosting algorithms emphasize samples with large gradients, without explicitly accounting for whether such errors originate from informative hard cases or from unreliable labels. We address this issue by reconsidering how sample reliability is evaluated during boosting. Instead of relying on instantaneous error, we examine the evolution of each sample's residuals across iterations. Based on this insight, we propose Information-Theoretic Trust Boosting (ITBoost), which uses the Minimum Description Length principle to measure the complexity of residual trajectories. Samples whose residual patterns fluctuate in an irregular manner are treated as less trustworthy and are down-weighted during learning. Theoretically, we derive a tighter generalization bound for ITBoost under label noise. Empirical results on various tabular benchmarks indicate that ITBoost provides improved robustness in noisy environments over leading boosting and deep tabular models, while retaining best average performance on clean data.