Diffusion-Promoted HDR Video Reconstruction

TL;DR

This paper introduces HDR-V-Diff, a diffusion-based method for HDR video reconstruction that captures realistic details and reduces artifacts by combining a latent diffusion model with temporal alignment and cross-attention mechanisms.

Contribution

The paper proposes a novel diffusion-promoted HDR video reconstruction framework with a latent diffusion model, temporal alignment, and cross-attention, improving quality and efficiency over existing methods.

Findings

Achieves state-of-the-art results on multiple datasets.

Effectively reduces ghosting artifacts and preserves details.

Incorporates a novel exposure embedding and zero-init cross-attention mechanism.

Abstract

High dynamic range (HDR) video reconstruction aims to generate HDR videos from low dynamic range (LDR) frames captured with alternating exposures. Most existing works solely rely on the regression-based paradigm, leading to adverse effects such as ghosting artifacts and missing details in saturated regions. In this paper, we propose a diffusion-promoted method for HDR video reconstruction, termed HDR-V-Diff, which incorporates a diffusion model to capture the HDR distribution. As such, HDR-V-Diff can reconstruct HDR videos with realistic details while alleviating ghosting artifacts. However, the direct introduction of video diffusion models would impose massive computational burden. Instead, to alleviate this burden, we first propose an HDR Latent Diffusion Model (HDR-LDM) to learn the distribution prior of single HDR frames. Specifically, HDR-LDM incorporates a tonemapping strategy to compress HDR frames into the latent space and a novel exposure embedding to aggregate the exposure information into the diffusion process. We then propose a Temporal-Consistent Alignment Module (TCAM) to learn the temporal information as a complement for HDR-LDM, which conducts coarse-to-fine feature alignment at different scales among video frames. Finally, we design a Zero-Init Cross-Attention (ZiCA) mechanism to effectively integrate the learned distribution prior and temporal information for generating HDR frames. Extensive experiments validate that HDR-V-Diff achieves state-of-the-art results on several representative datasets.