Online Overexposed Pixels Hallucination in Videos with Adaptive Reference Frame Selection

TL;DR

This paper introduces a learning-based, transformer-driven video HDR hallucination method that adaptively selects reference frames to reduce overexposure artifacts without complex HDR capture techniques.

Contribution

It presents a novel adaptive reference frame selection mechanism using reinforcement learning to improve HDR detail inference in videos with overexposed pixels.

Findings

Achieves state-of-the-art HDR hallucination quality

Effectively leverages temporal instabilities for better reconstruction

Operates causally without complex acquisition hardware

Abstract

Low dynamic range (LDR) cameras cannot deal with wide dynamic range inputs, frequently leading to local overexposure issues. We present a learning-based system to reduce these artifacts without resorting to complex acquisition mechanisms like alternating exposures or costly processing that are typical of high dynamic range (HDR) imaging. We propose a transformer-based deep neural network (DNN) to infer the missing HDR details. In an ablation study, we show the importance of using a multiscale DNN and train it with the proper cost function to achieve state-of-the-art quality. To aid the reconstruction of the overexposed areas, our DNN takes a reference frame from the past as an additional input. This leverages the commonly occurring temporal instabilities of autoexposure to our advantage: since well-exposed details in the current frame may be overexposed in the future, we use reinforcement learning to train a reference frame selection DNN that decides whether to adopt the current frame as a future reference. Without resorting to alternating exposures, we obtain therefore a causal, HDR hallucination algorithm with potential application in common video acquisition settings. Our demo video can be found at https://drive.google.com/file/d/1-r12BKImLOYCLUoPzdebnMyNjJ4Rk360/view