Covariance-free Partial Least Squares: An Incremental Dimensionality Reduction Method

TL;DR

This paper introduces CIPLS, an incremental, covariance-free version of PLS that efficiently learns low-dimensional data representations from streaming data, improving performance in vision tasks.

Contribution

It extends NIPALS for incremental processing, enabling PLS to be used on large and streaming datasets without requiring all data in memory.

Findings

Outperforms other incremental dimensionality reduction methods in face verification and image classification.

Achieves comparable feature selection results to state-of-the-art techniques.

Offers computational efficiency and preserves discriminative information.

Abstract

Dimensionality reduction plays an important role in computer vision problems since it reduces computational cost and is often capable of yielding more discriminative data representation. In this context, Partial Least Squares (PLS) has presented notable results in tasks such as image classification and neural network optimization. However, PLS is infeasible on large datasets, such as ImageNet, because it requires all the data to be in memory in advance, which is often impractical due to hardware limitations. Additionally, this requirement prevents us from employing PLS on streaming applications where the data are being continuously generated. Motivated by this, we propose a novel incremental PLS, named Covariance-free Incremental Partial Least Squares (CIPLS), which learns a low-dimensional representation of the data using a single sample at a time. In contrast to other state-of-the-art approaches, instead of adopting a partially-discriminative or SGD-based model, we extend Nonlinear Iterative Partial Least Squares (NIPALS) -- the standard algorithm used to compute PLS -- for incremental processing. Among the advantages of this approach are the preservation of discriminative information across all components, the possibility of employing its score matrices for feature selection, and its computational efficiency. We validate CIPLS on face verification and image classification tasks, where it outperforms several other incremental dimensionality reduction techniques. In the context of feature selection, CIPLS achieves comparable results when compared to state-of-the-art techniques.