Unsupervised Deformable Image Registration with Structural Nonparametric Smoothing

TL;DR

This paper introduces SmoothProper, a neural module that enforces smoothness and structural consistency in unsupervised deformable image registration, effectively handling large displacements and sparse features without hyperparameter tuning.

Contribution

We propose SmoothProper, a model-agnostic, plug-and-play module with a duality-based optimization layer that improves unsupervised DIR by propagating flow signals and enforcing smoothness during the forward pass.

Findings

Reduces registration error to 1.88 pixels on retinal images

First unsupervised DIR method to handle aperture and large-displacement challenges

Seamlessly integrates into existing frameworks with minimal parameters

Abstract

Learning-based deformable image registration (DIR) accelerates alignment by amortizing traditional optimization via neural networks. Label supervision further enhances accuracy, enabling efficient and precise nonlinear alignment of unseen scans. However, images with sparse features amid large smooth regions, such as retinal vessels, introduce aperture and large-displacement challenges that unsupervised DIR methods struggle to address. This limitation occurs because neural networks predict deformation fields in a single forward pass, leaving fields unconstrained post-training and shifting the regularization burden entirely to network weights. To address these issues, we introduce SmoothProper, a plug-and-play neural module enforcing smoothness and promoting message passing within the network's forward pass. By integrating a duality-based optimization layer with tailored interaction terms, SmoothProper efficiently propagates flow signals across spatial locations, enforces smoothness, and preserves structural consistency. It is model-agnostic, seamlessly integrates into existing registration frameworks with minimal parameter overhead, and eliminates regularizer hyperparameter tuning. Preliminary results on a retinal vessel dataset exhibiting aperture and large-displacement challenges demonstrate our method reduces registration error to 1.88 pixels on 2912x2912 images, marking the first unsupervised DIR approach to effectively address both challenges. The source code will be available at https://github.com/tinymilky/SmoothProper.