The Convergence of Dynamic Routing between Capsules

TL;DR

This paper provides a rigorous mathematical analysis and proof of convergence for the dynamic routing algorithm in capsule networks, addressing previous uncertainties about its behavior and effectiveness.

Contribution

It offers the first formal proof of convergence for dynamic routing in capsule networks and clarifies the algorithm's mathematical properties.

Findings

The objective function minimized by dynamic routing is concave.

Dynamic routing can be viewed as a nonlinear gradient method.

Convergence of the routing algorithm is mathematically proven.

Abstract

Capsule networks(CapsNet) are recently proposed neural network models with new processing layers, specifically for entity representation and discovery of images. It is well known that CapsNet have some advantages over traditional neural networks, especially in generalization capability. At the same time, some studies report negative experimental results. The causes of this contradiction have not been thoroughly analyzed. The preliminary experimental results show that the behavior of routing algorithms does not always produce good results as expected, and in most cases, different routing algorithms do not change the classification results, but simply polarize the link strength, especially when they continue to repeat without stopping. To realize the true potential of the CapsNet, deep mathematical analysis of the routing algorithms is crucial. In this paper, we will give the objective function that is minimized by the dynamic routing algorithm, which is a concave function. The dynamic routing algorithm can be regarded as nonlinear gradient method to solving an optimization algorithm under linear constraints, and its convergence can be strictly proved mathematically. Furthermore, the mathematically rigorous proof of the convergence is given for this class of iterative routing procedures. We analyze the relation between the objective function and the constraints solved by the dynamic routing algorithm in detail, and perform the corresponding routing experiment to analyze the effect of our convergence proof.