Deterministic Online Classification: Non-iteratively Reweighted Recursive Least-Squares for Binary Class Rebalancing

TL;DR

This paper introduces a new deterministic online classification algorithm based on reweighted least-squares that achieves constant time complexity, converges exactly to batch solutions, and outperforms existing methods on real-world data.

Contribution

The paper presents a novel online reweighted least-squares algorithm with constant time complexity for binary class rebalancing, ensuring exact convergence to batch solutions.

Findings

Achieves constant time complexity for online reweighting.

Exactly converges to batch reweighted least-squares solutions.

Outperforms existing stochastic online classification algorithms empirically.

Abstract

Deterministic solutions are becoming more critical for interpretability. Weighted Least-Squares (WLS) has been widely used as a deterministic batch solution with a specific weight design. In the online settings of WLS, exact reweighting is necessary to converge to its batch settings. In order to comply with its necessity, the iteratively reweighted least-squares algorithm is mainly utilized with a linearly growing time complexity which is not attractive for online learning. Due to the high and growing computational costs, an efficient online formulation of reweighted least-squares is desired. We introduce a new deterministic online classification algorithm of WLS with a constant time complexity for binary class rebalancing. We demonstrate that our proposed online formulation exactly converges to its batch formulation and outperforms existing state-of-the-art stochastic online binary classification algorithms in real-world data sets empirically.