Unsupervised model compression for multilayer bootstrap networks

TL;DR

This paper introduces a novel unsupervised model compression method that reduces the complexity of multilayer bootstrap networks (MBN) by using deep neural networks, maintaining accuracy while significantly improving prediction speed.

Contribution

It proposes a new framework for compressing unsupervised bootstrap models, specifically applying it to MBN with DNNs, achieving high accuracy and faster prediction.

Findings

Compressive MBN maintains high accuracy on MNIST.

Prediction speed is over a thousand times faster than original MBN.

Framework effectively combines unsupervised and supervised learning advantages.

Abstract

Recently, multilayer bootstrap network (MBN) has demonstrated promising performance in unsupervised dimensionality reduction. It can learn compact representations in standard data sets, i.e. MNIST and RCV1. However, as a bootstrap method, the prediction complexity of MBN is high. In this paper, we propose an unsupervised model compression framework for this general problem of unsupervised bootstrap methods. The framework compresses a large unsupervised bootstrap model into a small model by taking the bootstrap model and its application together as a black box and learning a mapping function from the input of the bootstrap model to the output of the application by a supervised learner. To specialize the framework, we propose a new technique, named compressive MBN. It takes MBN as the unsupervised bootstrap model and deep neural network (DNN) as the supervised learner. Our initial result on MNIST showed that compressive MBN not only maintains the high prediction accuracy of MBN but also is over thousands of times faster than MBN at the prediction stage. Our result suggests that the new technique integrates the effectiveness of MBN on unsupervised learning and the effectiveness and efficiency of DNN on supervised learning together for the effectiveness and efficiency of compressive MBN on unsupervised learning.