A Shallow High-Order Parametric Approach to Data Visualization and Compression

TL;DR

This paper introduces a simple yet effective shallow high-order parametric model for data visualization and compression, outperforming deep models in accuracy and efficiency, especially in kNN classification tasks.

Contribution

The paper presents a novel shallow high-order parametric embedding method (HOPE) that captures complex data interactions more efficiently than deep models, with new exemplar generation techniques for large datasets.

Findings

HOPE achieves a 0.65% test error on MNIST in 2D.

HOPE significantly speeds up kNN classification.

HOPE outperforms state-of-the-art deep embedding models.

Abstract

Explicit high-order feature interactions efficiently capture essential structural knowledge about the data of interest and have been used for constructing generative models. We present a supervised discriminative High-Order Parametric Embedding (HOPE) approach to data visualization and compression. Compared to deep embedding models with complicated deep architectures, HOPE generates more effective high-order feature mapping through an embarrassingly simple shallow model. Furthermore, two approaches to generating a small number of exemplars conveying high-order interactions to represent large-scale data sets are proposed. These exemplars in combination with the feature mapping learned by HOPE effectively capture essential data variations. Moreover, through HOPE, these exemplars are employed to increase the computational efficiency of kNN classification for fast information retrieval by thousands of times. For classification in two-dimensional embedding space on MNIST and USPS datasets, our shallow method HOPE with simple Sigmoid transformations significantly outperforms state-of-the-art supervised deep embedding models based on deep neural networks, and even achieved historically low test error rate of 0.65% in two-dimensional space on MNIST, which demonstrates the representational efficiency and power of supervised shallow models with high-order feature interactions.