# Training capsules as a routing-weighted product of expert neurons

## TL;DR

This paper introduces a novel unsupervised training method for capsule networks, modeling them as routing-weighted products of expert neurons, capable of generating realistic images from vision datasets.

## Contribution

It formulates capsule networks with dynamic routing as a product of expert neurons and develops a bottom-up unsupervised learning algorithm based on contrastive divergence.

## Key findings

- Able to generate realistic images from vision datasets
- Demonstrates the effectiveness of routing-weighted expert neuron modeling
- Provides a new unsupervised training approach for capsule networks

## Abstract

Capsules are the multidimensional analogue to scalar neurons in neural networks, and because they are multidimensional, much more complex routing schemes can be used to pass information forward through the network than what can be used in traditional neural networks. This work treats capsules as collections of neurons in a fully connected neural network, where sub-networks connecting capsules are weighted according to the routing coefficients determined by routing by agreement. An energy function is designed to reflect this model, and it follows that capsule networks with dynamic routing can be formulated as a product of expert neurons. By alternating between dynamic routing, which acts to both find subnetworks within the overall network as well as to mix the model distribution, and updating the parameters by the gradient of the contrastive divergence, a bottom-up, unsupervised learning algorithm is constructed for capsule networks with dynamic routing. The model and its training algorithm are qualitatively tested in the generative sense, and is able to produce realistic looking images from standard vision datasets.