On Data-Driven Saak Transform

TL;DR

This paper introduces a data-driven Saak transform that constructs hierarchical spatial-spectral features for pattern recognition without using labels or backpropagation, demonstrated on MNIST digit classification.

Contribution

The paper proposes a novel Saak transform framework that derives kernels from second-order statistics in a one-pass manner, avoiding supervised learning and enabling multi-stage spatial-spectral representations.

Findings

Saak transform effectively captures discriminant features for classification.

The method provides a reversible, multi-stage spatial-spectral feature extraction.

Demonstrated successful application on MNIST dataset.

Abstract

Being motivated by the multilayer RECOS (REctified-COrrelations on a Sphere) transform, we develop a data-driven Saak (Subspace approximation with augmented kernels) transform in this work. The Saak transform consists of three steps: 1) building the optimal linear subspace approximation with orthonormal bases using the second-order statistics of input vectors, 2) augmenting each transform kernel with its negative, 3) applying the rectified linear unit (ReLU) to the transform output. The Karhunen-Lo\'eve transform (KLT) is used in the first step. The integration of Steps 2 and 3 is powerful since they resolve the sign confusion problem, remove the rectification loss and allow a straightforward implementation of the inverse Saak transform at the same time. Multiple Saak transforms are cascaded to transform images of a larger size. All Saak transform kernels are derived from the second-order statistics of input random vectors in a one-pass feedforward manner. Neither data labels nor backpropagation is used in kernel determination. Multi-stage Saak transforms offer a family of joint spatial-spectral representations between two extremes; namely, the full spatial-domain representation and the full spectral-domain representation. We select Saak coefficients of higher discriminant power to form a feature vector for pattern recognition, and use the MNIST dataset classification problem as an illustrative example.