Why Capsule Neural Networks Do Not Scale: Challenging the Dynamic Parse-Tree Assumption

TL;DR

This paper critically examines capsule neural networks, revealing that their core parse-tree concept is absent in implementations and that they suffer from vanishing gradients, explaining their inability to scale to larger datasets.

Contribution

The paper provides a theoretical and experimental analysis showing the fundamental limitations of CapsNets, challenging their scalability and the core parse-tree assumption.

Findings

CapsNets lack the parse-tree structure they aim to implement.

CapsNets suffer from vanishing gradient problems during training.

These issues explain why CapsNets do not scale beyond toy datasets.

Abstract

Capsule neural networks replace simple, scalar-valued neurons with vector-valued capsules. They are motivated by the pattern recognition system in the human brain, where complex objects are decomposed into a hierarchy of simpler object parts. Such a hierarchy is referred to as a parse-tree. Conceptually, capsule neural networks have been defined to realize such parse-trees. The capsule neural network (CapsNet), by Sabour, Frosst, and Hinton, is the first actual implementation of the conceptual idea of capsule neural networks. CapsNets achieved state-of-the-art performance on simple image recognition tasks with fewer parameters and greater robustness to affine transformations than comparable approaches. This sparked extensive follow-up research. However, despite major efforts, no work was able to scale the CapsNet architecture to more reasonable-sized datasets. Here, we provide a reason for this failure and argue that it is most likely not possible to scale CapsNets beyond toy examples. In particular, we show that the concept of a parse-tree, the main idea behind capsule neuronal networks, is not present in CapsNets. We also show theoretically and experimentally that CapsNets suffer from a vanishing gradient problem that results in the starvation of many capsules during training.