Object-level Geometric Structure Preserving for Natural Image Stitching

TL;DR

This paper introduces a novel image stitching method that preserves object shapes at the object level using semantic segmentation and mesh transformations, improving both alignment and distortion control.

Contribution

It proposes OBJ-GSP, a new approach combining semantic segmentation and mesh-based transformations to maintain object structures during image stitching.

Findings

Outperforms existing methods in shape preservation and pixel alignment.

Demonstrates effectiveness in low-altitude aerial image stitching.

Introduces StitchBench, a comprehensive benchmark for image stitching.

Abstract

The topic of stitching images with globally natural structures holds paramount significance, with two main goals: pixel-level alignment and distortion prevention. The existing approaches exhibit the ability to align well, yet fall short in maintaining object structures. In this paper, we endeavour to safeguard the overall OBJect-level structures within images based on Global Similarity Prior (OBJ-GSP), on the basis of good alignment performance. Our approach leverages semantic segmentation models like the family of Segment Anything Model to extract the contours of any objects in a scene. Triangular meshes are employed in image transformation to protect the overall shapes of objects within images. The balance between alignment and distortion prevention is achieved by allowing the object meshes to strike a balance between similarity and projective transformation. We also demonstrate that object-level semantic information is necessary in low-altitude aerial image stitching. Additionally, we propose StitchBench, the largest image stitching benchmark with most diverse scenarios. Extensive experimental results demonstrate that OBJ-GSP outperforms existing methods in both pixel alignment and shape preservation. Code and dataset is publicly available at \url{https://github.com/RussRobin/OBJ-GSP}.