Auto-regressive transformation for image alignment

TL;DR

The paper introduces Auto-Regressive Transformation (ART), a novel iterative multi-scale method that improves image alignment accuracy in challenging conditions by focusing on critical regions and refining transformations hierarchically.

Contribution

It proposes a new auto-regressive, multi-scale approach with cross-attention guidance for robust image alignment, outperforming existing methods in difficult scenarios.

Findings

ART outperforms state-of-the-art methods on planar images.

Achieves comparable performance on 3D scene images.

Effectively handles feature-sparse regions and large deformations.

Abstract

Existing methods for image alignment struggle in cases involving feature-sparse regions, extreme scale and field-of-view differences, and large deformations, often resulting in suboptimal accuracy. Robustness to these challenges can be improved through iterative refinement of the transform field while focusing on critical regions in multi-scale image representations. We thus propose Auto-Regressive Transformation (ART), a novel method that iteratively estimates the coarse-to-fine transformations through an auto-regressive pipeline. Leveraging hierarchical multi-scale features, our network refines the transform field parameters using randomly sampled points at each scale. By incorporating guidance from the cross-attention layer, the model focuses on critical regions, ensuring accurate alignment even in challenging, feature-limited conditions. Extensive experiments demonstrate that ART significantly outperforms state-of-the-art methods on planar images and achieves comparable performance on 3D scene images, establishing it as a powerful and versatile solution for precise image alignment.