Adapting MIMO video restoration networks to low latency constraints

TL;DR

This paper enhances MIMO neural networks for low-latency video restoration by addressing performance drops and artifacts through recurrence and overlapping techniques, achieving state-of-the-art results and introducing a new drone footage benchmark.

Contribution

It introduces simple, adaptable solutions to improve low-latency MIMO video restoration networks by increasing temporal receptive field and smoothing transitions, with new state-of-the-art performance.

Findings

Improved reconstruction error in low-latency settings

Enhanced temporal consistency in video outputs

Effective solutions applicable to various architectures

Abstract

MIMO (multiple input, multiple output) approaches are a recent trend in neural network architectures for video restoration problems, where each network evaluation produces multiple output frames. The video is split into non-overlapping stacks of frames that are processed independently, resulting in a very appealing trade-off between output quality and computational cost. In this work we focus on the low-latency setting by limiting the number of available future frames. We find that MIMO architectures suffer from problems that have received little attention so far, namely (1) the performance drops significantly due to the reduced temporal receptive field, particularly for frames at the borders of the stack, (2) there are strong temporal discontinuities at stack transitions which induce a step-wise motion artifact. We propose two simple solutions to alleviate these problems: recurrence across MIMO stacks to boost the output quality by implicitly increasing the temporal receptive field, and overlapping of the output stacks to smooth the temporal discontinuity at stack transitions. These modifications can be applied to any MIMO architecture. We test them on three state-of-the-art video denoising networks with different computational cost. The proposed contributions result in a new state-of-the-art for low-latency networks, both in terms of reconstruction error and temporal consistency. As an additional contribution, we introduce a new benchmark consisting of drone footage that highlights temporal consistency issues that are not apparent in the standard benchmarks.