Generalized Low-Rank Update: Model Parameter Bounds for Low-Rank Training Data Modifications

TL;DR

This paper introduces the Generalized Low-Rank Update (GLRU), a novel method extending efficient model updates to a broad class of machine learning models beyond linear estimators, enabling quick adaptation to data modifications.

Contribution

The study develops GLRU, a framework that generalizes low-rank updates to regularized empirical risk minimization models like SVM and logistic regression, with efficient solution updates.

Findings

GLRU efficiently updates models with dataset changes.

Experimental results show improved speed in cross-validation and feature selection.

GLRU applies to a wide range of ML models beyond linear estimators.

Abstract

In this study, we have developed an incremental machine learning (ML) method that efficiently obtains the optimal model when a small number of instances or features are added or removed. This problem holds practical importance in model selection, such as cross-validation (CV) and feature selection. Among the class of ML methods known as linear estimators, there exists an efficient model update framework called the low-rank update that can effectively handle changes in a small number of rows and columns within the data matrix. However, for ML methods beyond linear estimators, there is currently no comprehensive framework available to obtain knowledge about the updated solution within a specific computational complexity. In light of this, our study introduces a method called the Generalized Low-Rank Update (GLRU) which extends the low-rank update framework of linear estimators to ML methods formulated as a certain class of regularized empirical risk minimization, including commonly used methods such as SVM and logistic regression. The proposed GLRU method not only expands the range of its applicability but also provides information about the updated solutions with a computational complexity proportional to the amount of dataset changes. To demonstrate the effectiveness of the GLRU method, we conduct experiments showcasing its efficiency in performing cross-validation and feature selection compared to other baseline methods.