Mitigating Vanishing Activations in Deep CapsNets Using Channel Pruning

TL;DR

This paper investigates the vanishing activation problem in deep Capsule Networks and proposes channel pruning as a method to mitigate it, enabling deeper networks with improved accuracy.

Contribution

It introduces a novel use of channel pruning to address vanishing activations in deep Capsule Networks, enhancing their scalability and performance.

Findings

Pruning reduces inactive capsules and improves accuracy.

Deeper capsule networks become feasible with pruning.

Channel pruning effectively mitigates vanishing activations.

Abstract

Capsule Networks outperform Convolutional Neural Networks in learning the part-whole relationships with viewpoint invariance, and the credit goes to their multidimensional capsules. It was assumed that increasing the number of capsule layers in the capsule networks would enhance the model performance. However, recent studies found that Capsule Networks lack scalability due to vanishing activations in the capsules of deeper layers. This paper thoroughly investigates the vanishing activation problem in deep Capsule Networks. To analyze this issue and understand how increasing capsule dimensions can facilitate deeper networks, various Capsule Network models are constructed and evaluated with different numbers of capsules, capsule dimensions, and intermediate layers for this paper. Unlike traditional model pruning, which reduces the number of model parameters and expedites model training, this study uses pruning to mitigate the vanishing activations in the deeper capsule layers. In addition, the backbone network and capsule layers are pruned with different pruning ratios to reduce the number of inactive capsules and achieve better model accuracy than the unpruned models.