Image Stitching Based on Planar Region Consensus

TL;DR

This paper introduces a novel image stitching method that leverages deep learning to extract planar regions from RGB images, enabling more accurate alignment without relying on global transformations, especially in challenging scenes.

Contribution

The paper proposes a new CNN-based approach for planar region segmentation and a mesh-based optimization framework for improved image stitching accuracy.

Findings

Outperforms state-of-the-art methods on challenging scenes.

Utilizes semantic information for precise planar segmentation.

Enables flexible local transformations for better alignment.

Abstract

Image stitching for two images without a global transformation between them is notoriously difficult. In this paper, noticing the importance of planar structure under perspective geometry, we propose a new image stitching method which stitches images by allowing for the alignment of a set of matched dominant planar regions. Clearly different from previous methods resorting to plane segmentation, the key to our approach is to utilize rich semantic information directly from RGB images to extract planar image regions with a deep Convolutional Neural Network (CNN). We specifically design a new module to make fully use of existing semantic segmentation networks to accommodate planar segmentation. To train the network, a dataset for planar region segmentation is contributed. With the planar region knowledge, a set of local transformations can be obtained by constraining matched regions, enabling more precise alignment in the overlapping area. We also use planar knowledge to estimate a transformation field over the whole image. The final mosaic is obtained by a mesh-based optimization framework which maintains high alignment accuracy and relaxes similarity transformation at the same time. Extensive experiments with quantitative comparisons show that our method can deal with different situations and outperforms the state-of-the-arts on challenging scenes.