Faster Principal Component Regression and Stable Matrix Chebyshev Approximation

TL;DR

This paper introduces a faster and more stable method for principal component regression that reduces computational complexity and does not require explicit principal component computation, suitable for large-scale problems.

Contribution

It presents a novel algorithm for PCR that uses fewer ridge regression calls and introduces a stable recurrence for matrix Chebyshev polynomials.

Findings

Reduces black-box calls from rac{}{}( ext) to rac{}{}( ext) for PCR.

Develops a stable recurrence formula for matrix Chebyshev polynomials.

Provides a degree-optimal polynomial approximation to the matrix sign function.

Abstract

We solve principal component regression (PCR), up to a multiplicative accuracy $1+\gamma$, by reducing the problem to $\tilde{O}(\gamma^{-1})$ black-box calls of ridge regression. Therefore, our algorithm does not require any explicit construction of the top principal components, and is suitable for large-scale PCR instances. In contrast, previous result requires $\tilde{O}(\gamma^{-2})$ such black-box calls. We obtain this result by developing a general stable recurrence formula for matrix Chebyshev polynomials, and a degree-optimal polynomial approximation to the matrix sign function. Our techniques may be of independent interests, especially when designing iterative methods.