Masked Capsule Autoencoders

TL;DR

This paper introduces Masked Capsule Autoencoders, a novel self-supervised pretraining method for Capsule Networks using masked image modelling, significantly improving their performance on complex, realistic datasets.

Contribution

It reformulates Capsule Networks to incorporate masked image modelling pretraining, achieving state-of-the-art results and demonstrating the benefits of self-supervised learning for capsules.

Findings

Capsule Networks benefit from self-supervised pretraining.

Achieved 9% improvement on Imagenette dataset.

State-of-the-art results for Capsule Networks on complex images.

Abstract

We propose Masked Capsule Autoencoders (MCAE), the first Capsule Network that utilises pretraining in a modern self-supervised paradigm, specifically the masked image modelling framework. Capsule Networks have emerged as a powerful alternative to Convolutional Neural Networks (CNNs). They have shown favourable properties when compared to Vision Transformers (ViT), but have struggled to effectively learn when presented with more complex data. This has led to Capsule Network models that do not scale to modern tasks. Our proposed MCAE model alleviates this issue by reformulating the Capsule Network to use masked image modelling as a pretraining stage before finetuning in a supervised manner. Across several experiments and ablations studies we demonstrate that similarly to CNNs and ViTs, Capsule Networks can also benefit from self-supervised pretraining, paving the way for further advancements in this neural network domain. For instance, by pretraining on the Imagenette dataset-consisting of 10 classes of Imagenet-sized images-we achieve state-of-the-art results for Capsule Networks, demonstrating a 9% improvement compared to our baseline model. Thus, we propose that Capsule Networks benefit from and should be trained within a masked image modelling framework, using a novel capsule decoder, to enhance a Capsule Network's performance on realistically sized images.