SDRTV-to-HDRTV Conversion via Spatial-Temporal Feature Fusion

TL;DR

This paper introduces DSLNet, a neural network that fuses spatial-temporal features for converting SDR videos to HDR, significantly improving quality by leveraging multi-frame information.

Contribution

The paper proposes a novel multi-frame fusion neural network with dynamic alignment, feature modulation, and quality enhancement modules for SDR to HDR video conversion.

Findings

Achieves state-of-the-art performance on HDR video conversion

Constructs a multi-frame HDR dataset for evaluation

Demonstrates effectiveness of spatial-temporal feature fusion

Abstract

HDR(High Dynamic Range) video can reproduce realistic scenes more realistically, with a wider gamut and broader brightness range. HDR video resources are still scarce, and most videos are still stored in SDR (Standard Dynamic Range) format. Therefore, SDRTV-to-HDRTV Conversion (SDR video to HDR video) can significantly enhance the user's video viewing experience. Since the correlation between adjacent video frames is very high, the method utilizing the information of multiple frames can improve the quality of the converted HDRTV. Therefore, we propose a multi-frame fusion neural network \textbf{DSLNet} for SDRTV to HDRTV conversion. We first propose a dynamic spatial-temporal feature alignment module \textbf{DMFA}, which can align and fuse multi-frame. Then a novel spatial-temporal feature modulation module \textbf{STFM}, STFM extracts spatial-temporal information of adjacent frames for more accurate feature modulation. Finally, we design a quality enhancement module \textbf{LKQE} with large kernels, which can enhance the quality of generated HDR videos. To evaluate the performance of the proposed method, we construct a corresponding multi-frame dataset using HDR video of the HDR10 standard to conduct a comprehensive evaluation of different methods. The experimental results show that our method obtains state-of-the-art performance. The dataset and code will be released.