Sparse Unsupervised Capsules Generalize Better

TL;DR

This paper demonstrates that unsupervised sparse training of capsule networks enhances their generalization and preserves desirable properties, enabling deeper architectures and significantly improving classification accuracy on affNIST.

Contribution

It introduces a method for unsupervised sparsening of capsule layers that restores capsule qualities and improves generalization, surpassing supervised masking approaches.

Findings

Accuracy on affNIST improved from 79% to 90%.

Unsupervised sparsening restores capsule qualities and enables deeper networks.

Supervised masking limits capsule network depth and qualities.

Abstract

We show that unsupervised training of latent capsule layers using only the reconstruction loss, without masking to select the correct output class, causes a loss of equivariances and other desirable capsule qualities. This implies that supervised capsules networks can't be very deep. Unsupervised sparsening of latent capsule layer activity both restores these qualities and appears to generalize better than supervised masking, while potentially enabling deeper capsules networks. We train a sparse, unsupervised capsules network of similar geometry to Sabour et al (2017) on MNIST, and then test classification accuracy on affNIST using an SVM layer. Accuracy is improved from benchmark 79% to 90%.