GBM-based Bregman Proximal Algorithms for Constrained Learning

TL;DR

This paper introduces a Bregman proximal algorithm framework for constrained learning with gradient boosting machines, enabling effective handling of complex risk constraints in applications like Neyman-Pearson classification and fairness, while maintaining compatibility with popular GBM implementations.

Contribution

It develops a novel Bregman primal-dual method for constrained learning that integrates with existing GBM tools and guarantees global optimality for convex cases.

Findings

The proposed algorithm effectively handles complex risk constraints.

It seamlessly integrates with popular GBM implementations like XGBoost and LightGBM.

Experimental results demonstrate the framework's effectiveness across applications.

Abstract

As the complexity of learning tasks surges, modern machine learning encounters a new constrained learning paradigm characterized by more intricate and data-driven function constraints. Prominent applications include Neyman-Pearson classification (NPC) and fairness classification, which entail specific risk constraints that render standard projection-based training algorithms unsuitable. Gradient boosting machines (GBMs) are among the most popular algorithms for supervised learning; however, they are generally limited to unconstrained settings. In this paper, we adapt the GBM for constrained learning tasks within the framework of Bregman proximal algorithms. We introduce a new Bregman primal-dual method with a global optimality guarantee when the learning objective and constraint functions are convex. In cases of nonconvex functions, we demonstrate how our algorithm remains effective under a Bregman proximal point framework. Distinct from existing constrained learning algorithms, ours possess a unique advantage in their ability to seamlessly integrate with publicly available GBM implementations such as XGBoost (Chen and Guestrin, 2016) and LightGBM (Ke et al., 2017), exclusively relying on their public interfaces. We provide substantial experimental evidence to showcase the effectiveness of the Bregman algorithm framework. While our primary focus is on NPC and fairness ML, our framework holds significant potential for a broader range of constrained learning applications. The source code is currently freely available at https://github.com/zhenweilin/ConstrainedGBM}{https://github.com/zhenweilin/ConstrainedGBM.