Can representation learning for multimodal image registration be improved by supervision of intermediate layers?

TL;DR

This study investigates whether adding supervision to intermediate layers in contrastive learning improves multimodal image registration, finding that it often does not and can cause embedding space collapse.

Contribution

The paper evaluates the impact of intermediate layer supervision in contrastive learning for multimodal registration, revealing it may hinder rather than help registration performance.

Findings

Supervision on intermediate layers does not improve registration accuracy.

Additional supervision can cause partial dimensional collapse of embeddings.

Representations learned without extra supervision perform best in registration tasks.

Abstract

Multimodal imaging and correlative analysis typically require image alignment. Contrastive learning can generate representations of multimodal images, reducing the challenging task of multimodal image registration to a monomodal one. Previously, additional supervision on intermediate layers in contrastive learning has improved biomedical image classification. We evaluate if a similar approach improves representations learned for registration to boost registration performance. We explore three approaches to add contrastive supervision to the latent features of the bottleneck layer in the U-Nets encoding the multimodal images and evaluate three different critic functions. Our results show that representations learned without additional supervision on latent features perform best in the downstream task of registration on two public biomedical datasets. We investigate the performance drop by exploiting recent insights in contrastive learning in classification and self-supervised learning. We visualize the spatial relations of the learned representations by means of multidimensional scaling, and show that additional supervision on the bottleneck layer can lead to partial dimensional collapse of the intermediate embedding space.