PipeFlow: Pipelined Processing and Motion-Aware Frame Selection for Long-Form Video Editing

TL;DR

PipeFlow introduces a scalable, motion-aware, pipelined approach for long-form video editing that significantly reduces computational costs and speeds up processing by dividing videos into segments and interpolating skipped frames.

Contribution

The paper presents a novel pipelined video editing method that combines motion analysis, parallel processing, and frame interpolation to enable efficient editing of arbitrarily long videos.

Findings

Achieves up to 9.6X speedup over TokenFlow.

Achieves up to 31.7X speedup over DMT.

Enables linear scaling of editing time with video length.

Abstract

Long-form video editing poses unique challenges due to the exponential increase in the computational cost from joint editing and Denoising Diffusion Implicit Models (DDIM) inversion across extended sequences. To address these limitations, we propose PipeFlow, a scalable, pipelined video editing method that introduces three key innovations: First, based on a motion analysis using Structural Similarity Index Measure (SSIM) and Optical Flow, we identify and propose to skip editing of frames with low motion. Second, we propose a pipelined task scheduling algorithm that splits a video into multiple segments and performs DDIM inversion and joint editing in parallel based on available GPU memory. Lastly, we leverage a neural network-based interpolation technique to smooth out the border frames between segments and interpolate the previously skipped frames. Our method uniquely scales to longer videos by dividing them into smaller segments, allowing PipeFlow's editing time to increase linearly with video length. In principle, this enables editing of infinitely long videos without the growing per-frame computational overhead encountered by other methods. PipeFlow achieves up to a 9.6X speedup compared to TokenFlow and a 31.7X speedup over Diffusion Motion Transfer (DMT).