On Convolutional Approximations to Linear Dimensionality Reduction Operators for Large Scale Data Processing

TL;DR

This paper investigates the approximation of linear dimensionality reduction operators using partial circulant matrices, revealing limitations for large matrices and proposing a new approximation method with a sparse factorization technique.

Contribution

It demonstrates the limitations of partial circulant approximations for large LDR matrices and introduces a novel sparse matrix factorization method for improved approximations.

Findings

Most large LDR matrices cannot be well approximated by partial circulant matrices.

A generalized approximation framework using a product of a larger partial circulant matrix and a post-processing matrix.

Preliminary evidence shows the potential of the proposed sparse factorization approach.

Abstract

In this paper, we examine the problem of approximating a general linear dimensionality reduction (LDR) operator, represented as a matrix $A \in \mathbb{R}^{m \times n}$ with $m < n$, by a partial circulant matrix with rows related by circular shifts. Partial circulant matrices admit fast implementations via Fourier transform methods and subsampling operations; our investigation here is motivated by a desire to leverage these potential computational improvements in large-scale data processing tasks. We establish a fundamental result, that most large LDR matrices (whose row spaces are uniformly distributed) in fact cannot be well approximated by partial circulant matrices. Then, we propose a natural generalization of the partial circulant approximation framework that entails approximating the range space of a given LDR operator $A$ over a restricted domain of inputs, using a matrix formed as a product of a partial circulant matrix having $m '> m$ rows and a $m \times k$ 'post processing' matrix. We introduce a novel algorithmic technique, based on sparse matrix factorization, for identifying the factors comprising such approximations, and provide preliminary evidence to demonstrate the potential of this approach.