Fast Swap-Based Element Selection for Multiplication-Free Dimension Reduction

TL;DR

This paper introduces a fast, multiplication-free element selection algorithm for dimension reduction, using swap-based local search to efficiently identify the most informative elements, demonstrated on MNIST data.

Contribution

It proposes a novel, efficient swap-based algorithm for element selection in dimension reduction that avoids matrix multiplications, suitable for resource-constrained systems.

Findings

Effective dimension reduction on MNIST dataset

Reduces computational cost by eliminating multiplications

Achieves comparable performance to traditional methods

Abstract

In this paper, we propose a fast algorithm for element selection, a multiplication-free form of dimension reduction that produces a dimension-reduced vector by simply selecting a subset of elements from the input. Dimension reduction is a fundamental technique for reducing unnecessary model parameters, mitigating overfitting, and accelerating training and inference. A standard approach is principal component analysis (PCA), but PCA relies on matrix multiplications; on resource-constrained systems, the multiplication count itself can become a bottleneck. Element selection eliminates this cost because the reduction consists only of selecting elements, and thus the key challenge is to determine which elements should be retained. We evaluate a candidate subset through the minimum mean-squared error of linear regression that predicts a target vector from the selected elements, where the target may be, for example, a one-hot label vector in classification. When an explicit target is unavailable, the input itself can be used as the target, yielding a reconstruction-based criterion. The resulting optimization is combinatorial, and exhaustive search is impractical. To address this, we derive an efficient formula for the objective change caused by swapping a selected and an unselected element, using the matrix inversion lemma, and we perform a swap-based local search that repeatedly applies objective-decreasing swaps until no further improvement is possible. Experiments on MNIST handwritten-digit images demonstrate the effectiveness of the proposed method.