Bringing Old Films Back to Life

TL;DR

This paper introduces a recurrent transformer network (RTN) that leverages adjacent frames and hidden knowledge to restore heavily degraded old films, ensuring temporal consistency and effective defect localization.

Contribution

The novel RTN framework uses a recurrent transformer architecture to improve old film restoration by exploiting temporal information and unsupervised defect localization.

Findings

RTN outperforms existing methods on synthetic and real-world datasets.

The framework effectively propagates color from keyframes to entire videos.

Transformer blocks enhance spatial restoration and handle mixed degradations.

Abstract

We present a learning-based framework, recurrent transformer network (RTN), to restore heavily degraded old films. Instead of performing frame-wise restoration, our method is based on the hidden knowledge learned from adjacent frames that contain abundant information about the occlusion, which is beneficial to restore challenging artifacts of each frame while ensuring temporal coherency. Moreover, contrasting the representation of the current frame and the hidden knowledge makes it possible to infer the scratch position in an unsupervised manner, and such defect localization generalizes well to real-world degradations. To better resolve mixed degradation and compensate for the flow estimation error during frame alignment, we propose to leverage more expressive transformer blocks for spatial restoration. Experiments on both synthetic dataset and real-world old films demonstrate the significant superiority of the proposed RTN over existing solutions. In addition, the same framework can effectively propagate the color from keyframes to the whole video, ultimately yielding compelling restored films. The implementation and model will be released at https://github.com/raywzy/Bringing-Old-Films-Back-to-Life.