Fast and Accurate: Video Enhancement using Sparse Depth

TL;DR

This paper introduces a fast, accurate video enhancement framework that leverages sparse flow estimation combining point cloud and IMU data, outperforming dense flow methods in speed and quality.

Contribution

It proposes a lightweight sparse flow estimation method that integrates sparse data sources for high-quality video enhancement, reducing computational cost.

Findings

Achieves up to 187.41x speedup over existing methods.

Provides 0.42 to 6.70 dB quality improvement.

Demonstrates versatility across super-resolution, deblurring, and denoising.

Abstract

This paper presents a general framework to build fast and accurate algorithms for video enhancement tasks such as super-resolution, deblurring, and denoising. Essential to our framework is the realization that the accuracy, rather than the density, of pixel flows is what is required for high-quality video enhancement. Most of prior works take the opposite approach: they estimate dense (per-pixel)-but generally less robust-flows, mostly using computationally costly algorithms. Instead, we propose a lightweight flow estimation algorithm; it fuses the sparse point cloud data and (even sparser and less reliable) IMU data available in modern autonomous agents to estimate the flow information. Building on top of the flow estimation, we demonstrate a general framework that integrates the flows in a plug-and-play fashion with different task-specific layers. Algorithms built in our framework achieve 1.78x - 187.41x speedup while providing a 0.42 dB - 6.70 dB quality improvement over competing methods.