# Do Subsampled Newton Methods Work for High-Dimensional Data?

## TL;DR

This paper provides a theoretical justification for the effectiveness of subsampled Newton methods in high-dimensional settings, showing they require significantly fewer samples than previously thought, especially when leveraging ridge leverage scores.

## Contribution

It proves that only a small number of samples based on ridge leverage scores are needed for Hessian approximation in high-dimensional data, extending applicability to distributed and non-smooth problems.

## Key findings

- Subsampled Newton methods need only ^ff_ff samples for Hessian approximation.
- The method is effective even when data dimension is large and comparable to data size.
- Extensions to distributed and non-smooth regularized optimization problems are provided.

## Abstract

Subsampled Newton methods approximate Hessian matrices through subsampling techniques, alleviating the cost of forming Hessian matrices but using sufficient curvature information. However, previous results require $\Omega (d)$ samples to approximate Hessians, where $d$ is the dimension of data points, making it less practically feasible for high-dimensional data. The situation is deteriorated when $d$ is comparably as large as the number of data points $n$, which requires to take the whole dataset into account, making subsampling useless. This paper theoretically justifies the effectiveness of subsampled Newton methods on high dimensional data. Specifically, we prove only $\widetilde{\Theta}(d^\gamma_{\rm eff})$ samples are needed in the approximation of Hessian matrices, where $d^\gamma_{\rm eff}$ is the $\gamma$-ridge leverage and can be much smaller than $d$ as long as $n\gamma \gg 1$. Additionally, we extend this result so that subsampled Newton methods can work for high-dimensional data on both distributed optimization problems and non-smooth regularized problems.