Robust image stitching with multiple registrations

TL;DR

This paper introduces a robust image stitching method that employs multiple registrations to improve panorama quality, especially in scenes with depth variation or motion, by extending seam finding techniques with new energy terms.

Contribution

It proposes using multiple registrations in panorama creation and modifies seam finding energy functions to handle duplication and tearing issues.

Findings

Significantly better panoramas in scenes with motion or parallax.

Extends seam finding with new energy terms for multiple registrations.

Improves accuracy in scenes with depth variation or object motion.

Abstract

Panorama creation is one of the most widely deployed techniques in computer vision. In addition to industry applications such as Google Street View, it is also used by millions of consumers in smartphones and other cameras. Traditionally, the problem is decomposed into three phases: registration, which picks a single transformation of each source image to align it to the other inputs, seam finding, which selects a source image for each pixel in the final result, and blending, which fixes minor visual artifacts. Here, we observe that the use of a single registration often leads to errors, especially in scenes with significant depth variation or object motion. We propose instead the use of multiple registrations, permitting regions of the image at different depths to be captured with greater accuracy. MRF inference techniques naturally extend to seam finding over multiple registrations, and we show here that their energy functions can be readily modified with new terms that discourage duplication and tearing, common problems that are exacerbated by the use of multiple registrations. Our techniques are closely related to layer-based stereo, and move image stitching closer to explicit scene modeling. Experimental evidence demonstrates that our techniques often generate significantly better panoramas when there is substantial motion or parallax.