StabStitch++: Unsupervised Online Video Stitching with Spatiotemporal Bidirectional Warps

TL;DR

StabStitch++ introduces an unsupervised online video stitching framework that simultaneously achieves spatial alignment and temporal stabilization, effectively reducing warping shake and enhancing visual quality in real-time applications.

Contribution

The paper proposes a novel bidirectional warping approach with a differentiable decomposition module and a warp smoothing model for unsupervised, real-time video stitching with stabilization.

Findings

Outperforms existing methods in stitching quality and robustness.

Effectively reduces warping shake in stitched videos.

Enables real-time online video stitching with improved stability.

Abstract

We retarget video stitching to an emerging issue, named warping shake, which unveils the temporal content shakes induced by sequentially unsmooth warps when extending image stitching to video stitching. Even if the input videos are stable, the stitched video can inevitably cause undesired warping shakes and affect the visual experience. To address this issue, we propose StabStitch++, a novel video stitching framework to realize spatial stitching and temporal stabilization with unsupervised learning simultaneously. First, different from existing learning-based image stitching solutions that typically warp one image to align with another, we suppose a virtual midplane between original image planes and project them onto it. Concretely, we design a differentiable bidirectional decomposition module to disentangle the homography transformation and incorporate it into our spatial warp, evenly spreading alignment burdens and projective distortions across two views. Then, inspired by camera paths in video stabilization, we derive the mathematical expression of stitching trajectories in video stitching by elaborately integrating spatial and temporal warps. Finally, a warp smoothing model is presented to produce stable stitched videos with a hybrid loss to simultaneously encourage content alignment, trajectory smoothness, and online collaboration. Compared with StabStitch that sacrifices alignment for stabilization, StabStitch++ makes no compromise and optimizes both of them simultaneously, especially in the online mode. To establish an evaluation benchmark and train the learning framework, we build a video stitching dataset with a rich diversity in camera motions and scenes. Experiments exhibit that StabStitch++ surpasses current solutions in stitching performance, robustness, and efficiency, offering compelling advancements in this field by building a real-time online video stitching system.