Modification Takes Courage: Seamless Image Stitching via Reference-Driven Inpainting

TL;DR

This paper introduces RDIStitcher, a reference-driven inpainting approach for seamless image stitching that leverages self-supervised training and multimodal metrics to improve content coherence and generalization in challenging scenarios.

Contribution

The paper presents a novel reference-driven inpainting model for image stitching, combined with self-supervised training and new quality metrics, advancing the state-of-the-art in seamless image fusion.

Findings

Significantly improves content coherence in stitched images.

Demonstrates strong zero-shot generalization capabilities.

Outperforms existing methods in challenging scenarios.

Abstract

Current image stitching methods often produce noticeable seams in challenging scenarios such as uneven hue and large parallax. To tackle this problem, we propose the Reference-Driven Inpainting Stitcher (RDIStitcher), which reformulates the image fusion and rectangling as a reference-based inpainting model, incorporating a larger modification fusion area and stronger modification intensity than previous methods. Furthermore, we introduce a self-supervised model training method, which enables the implementation of RDIStitcher without requiring labeled data by fine-tuning a Text-to-Image (T2I) diffusion model. Recognizing difficulties in assessing the quality of stitched images, we present the Multimodal Large Language Models (MLLMs)-based metrics, offering a new perspective on evaluating stitched image quality. Compared to the state-of-the-art (SOTA) method, extensive experiments demonstrate that our method significantly enhances content coherence and seamless transitions in the stitched images. Especially in the zero-shot experiments, our method exhibits strong generalization capabilities. Code: https://github.com/yayoyo66/RDIStitcher