Impact of Disentanglement on Pruning Neural Networks

TL;DR

This paper investigates how learning disentangled representations with Beta-VAE affects neural network pruning efficiency for classification tasks on MNIST and CIFAR10.

Contribution

It explores the impact of disentanglement on pruning processes, providing insights and future directions for model compression techniques.

Findings

Disentangled representations influence pruning effectiveness.

Experiments conducted on MNIST and CIFAR10 datasets.

Challenges in achieving disentanglement are identified.

Abstract

Deploying deep learning neural networks on edge devices, to accomplish task specific objectives in the real-world, requires a reduction in their memory footprint, power consumption, and latency. This can be realized via efficient model compression. Disentangled latent representations produced by variational autoencoder (VAE) networks are a promising approach for achieving model compression because they mainly retain task-specific information, discarding useless information for the task at hand. We make use of the Beta-VAE framework combined with a standard criterion for pruning to investigate the impact of forcing the network to learn disentangled representations on the pruning process for the task of classification. In particular, we perform experiments on MNIST and CIFAR10 datasets, examine disentanglement challenges, and propose a path forward for future works.