Harmonized Gradient Descent for Class Imbalanced Data Stream Online Learning

TL;DR

This paper introduces the harmonized gradient descent (HGD) algorithm to improve online learning from imbalanced data streams by balancing gradient norms, leading to better minority class learning without extra data or parameters.

Contribution

The paper proposes a novel gradient-based method, HGD, that balances gradient norms across classes, enhancing imbalanced data stream learning without additional data buffers or parameters.

Findings

HGD achieves sub-linear regret bounds under mild assumptions.

HGD outperforms existing online imbalance learning methods.

Experimental results confirm HGD's efficiency and effectiveness.

Abstract

Many real-world data are sequentially collected over time and often exhibit skewed class distributions, resulting in imbalanced data streams. While existing approaches have explored several strategies, such as resampling and reweighting, for imbalanced data stream learning, our work distinguishes itself by addressing the imbalance problem through training modification, particularly focusing on gradient descent techniques. We introduce the harmonized gradient descent (HGD) algorithm, which aims to equalize the norms of gradients across different classes. By ensuring the gradient norm balance, HGD mitigates under-fitting for minor classes and achieves balanced online learning. Notably, HGD operates in a streamlined implementation process, requiring no data-buffer, extra parameters, or prior knowledge, making it applicable to any learning models utilizing gradient descent for optimization. Theoretical analysis, based on a few common and mild assumptions, shows that HGD achieves a satisfied sub-linear regret bound. The proposed algorithm are compared with the commonly used online imbalance learning methods under several imbalanced data stream scenarios. Extensive experimental evaluations demonstrate the efficiency and effectiveness of HGD in learning imbalanced data streams.