Approximate Nearest Neighbor Fields in Video

TL;DR

RIANN is a real-time algorithm for approximate nearest neighbor search in video patches, leveraging temporal coherence to achieve high speed and enabling various real-time video processing applications.

Contribution

The paper introduces RIANN, a novel real-time ANN search algorithm optimized for video, significantly faster than previous methods and suitable for multiple real-time video tasks.

Findings

RIANN is up to 100 times faster than previous ANN methods.

It operates in real-time for typical video sequences.

It enables real-time applications like colorization and denoising.

Abstract

We introduce RIANN (Ring Intersection Approximate Nearest Neighbor search), an algorithm for matching patches of a video to a set of reference patches in real-time. For each query, RIANN finds potential matches by intersecting rings around key points in appearance space. Its search complexity is reversely correlated to the amount of temporal change, making it a good fit for videos, where typically most patches change slowly with time. Experiments show that RIANN is up to two orders of magnitude faster than previous ANN methods, and is the only solution that operates in real-time. We further demonstrate how RIANN can be used for real-time video processing and provide examples for a range of real-time video applications, including colorization, denoising, and several artistic effects.