On Compression of Unsupervised Neural Nets by Pruning Weak Connections
Zhiwen Zuo, Lei Zhao, Liwen Zuo, Feng Jiang, Wei Xing, Dongming Lu
TL;DR
This paper presents a method to significantly compress unsupervised neural networks like RBMs and DBNs by pruning weak connections, maintaining performance while reducing parameters.
Contribution
It introduces a novel pruning technique for RBMs and an unsupervised sparse architecture selection algorithm for deep networks, enabling effective compression.
Findings
Parameter reduction with minimal performance loss
Sparse deep networks maintain generative and discriminative capabilities
Method applicable to various unsupervised neural architectures
Abstract
Unsupervised neural nets such as Restricted Boltzmann Machines(RBMs) and Deep Belif Networks(DBNs), are powerful in automatic feature extraction,unsupervised weight initialization and density estimation. In this paper,we demonstrate that the parameters of these neural nets can be dramatically reduced without affecting their performance. We describe a method to reduce the parameters required by RBM which is the basic building block for deep architectures. Further we propose an unsupervised sparse deep architectures selection algorithm to form sparse deep neural networks.Experimental results show that there is virtually no loss in either generative or discriminative performance.
Click any figure to enlarge with its caption.
Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Figure 6
Figure 7
Figure 8
Figure 9Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsGenerative Adversarial Networks and Image Synthesis · Music and Audio Processing · Neural Networks and Applications