Capsule Networks with Max-Min Normalization

TL;DR

This paper introduces Max-Min normalization for Capsule Networks, replacing Softmax, leading to improved accuracy and stability across multiple datasets, and enabling more routing iterations without performance loss.

Contribution

The paper proposes Max-Min normalization as a novel alternative to Softmax in Capsule Networks, enhancing routing flexibility and accuracy.

Findings

Max-Min normalization improves test accuracy on five datasets.

Max-Min allows more routing iterations without performance decline.

CapsNet with Max-Min achieves 0.20% error on MNIST.

Abstract

Capsule Networks (CapsNet) use the Softmax function to convert the logits of the routing coefficients into a set of normalized values that signify the assignment probabilities between capsules in adjacent layers. We show that the use of Softmax prevents capsule layers from forming optimal couplings between lower and higher-level capsules. Softmax constrains the dynamic range of the routing coefficients and leads to probabilities that remain mostly uniform after several routing iterations. Instead, we propose the use of Max-Min normalization. Max-Min performs a scale-invariant normalization of the logits that allows each lower-level capsule to take on an independent value, constrained only by the bounds of normalization. Max-Min provides consistent improvement in test accuracy across five datasets and allows more routing iterations without a decrease in network performance. A single CapsNet trained using Max-Min achieves an improved test error of 0.20% on the MNIST dataset. With a simple 3-model majority vote, we achieve a test error of 0.17% on MNIST.