Fast Second-Order Online Kernel Learning through Incremental Matrix Sketching and Decomposition

TL;DR

FORKS introduces an efficient incremental matrix sketching and decomposition method for second-order online kernel learning, significantly reducing computational complexity while maintaining high predictive performance in streaming environments.

Contribution

The paper presents FORKS, a novel incremental approach that improves second-order OKL scalability and robustness through matrix sketching and decomposition techniques.

Findings

FORKS achieves logarithmic regret guarantees similar to existing methods.

FORKS operates with linear time complexity relative to the budget.

Experimental results show superior scalability and robustness in real-world datasets.

Abstract

Online Kernel Learning (OKL) has attracted considerable research interest due to its promising predictive performance in streaming environments. Second-order approaches are particularly appealing for OKL as they often offer substantial improvements in regret guarantees. However, existing second-order OKL approaches suffer from at least quadratic time complexity with respect to the pre-set budget, rendering them unsuitable for meeting the real-time demands of large-scale streaming recommender systems. The singular value decomposition required to obtain explicit feature mapping is also computationally expensive due to the complete decomposition process. Moreover, the absence of incremental updates to manage approximate kernel space causes these algorithms to perform poorly in adversarial environments and real-world streaming recommendation datasets. To address these issues, we propose FORKS, a fast incremental matrix sketching and decomposition approach tailored for second-order OKL. FORKS constructs an incremental maintenance paradigm for second-order kernelized gradient descent, which includes incremental matrix sketching for kernel approximation and incremental matrix decomposition for explicit feature mapping construction. Theoretical analysis demonstrates that FORKS achieves a logarithmic regret guarantee on par with other second-order approaches while maintaining a linear time complexity w.r.t. the budget, significantly enhancing efficiency over existing approaches. We validate the performance of FORKS through extensive experiments conducted on real-world streaming recommendation datasets, demonstrating its superior scalability and robustness against adversarial attacks.