An hybrid stochastic Newton algorithm for logistic regression

TL;DR

This paper introduces a hybrid stochastic Newton algorithm for logistic regression that combines Hessian and gradient information, ensuring convergence and improving convergence rates for large-scale binary classification tasks.

Contribution

The paper proposes a novel hybrid stochastic Newton algorithm that integrates Hessian and gradient estimates, with proven convergence and enhanced convergence rates.

Findings

Proves almost sure convergence to the true parameter.

Establishes a central limit theorem for the algorithm.

Shows almost sure convergence of the cumulative excess risk.

Abstract

In this paper, we investigate a second-order stochastic algorithm for solving large-scale binary classification problems. We propose to make use of a new hybrid stochastic Newton algorithm that includes two weighted components in the Hessian matrix estimation: the first one coming from the natural Hessian estimate and the second associated with the stochastic gradient information. Our motivation comes from the fact that both parts evaluated at the true parameter of logistic regression, are equal to the Hessian matrix. This new formulation has several advantages and it enables us to prove the almost sure convergence of our stochastic algorithm to the true parameter. Moreover, we significantly improve the almost sure rate of convergence to the Hessian matrix. Furthermore, we establish the central limit theorem for our hybrid stochastic Newton algorithm. Finally, we show a surprising result on the almost sure convergence of the cumulative excess risk.