SC-HVPPNet: Spatial and Channel Hybrid-Attention Video Post-Processing Network with CNN and Transformer

TL;DR

This paper introduces SC-HVPPNet, a novel video post-processing network that effectively combines CNN and Transformer features through spatial and channel attention modules, significantly improving video restoration quality.

Contribution

The paper proposes a hybrid attention framework that enhances CNN-Transformer interaction for video post-processing, with novel spatial and channel attention fusion modules.

Findings

Boosts video restoration quality significantly.

Achieves average bitrate savings of over 5% in Y, U, V components.

Demonstrates effective cooperation between local and global features.

Abstract

Convolutional Neural Network (CNN) and Transformer have attracted much attention recently for video post-processing (VPP). However, the interaction between CNN and Transformer in existing VPP methods is not fully explored, leading to inefficient communication between the local and global extracted features. In this paper, we explore the interaction between CNN and Transformer in the task of VPP, and propose a novel Spatial and Channel Hybrid-Attention Video Post-Processing Network (SC-HVPPNet), which can cooperatively exploit the image priors in both spatial and channel domains. Specifically, in the spatial domain, a novel spatial attention fusion module is designed, in which two attention weights are generated to fuse the local and global representations collaboratively. In the channel domain, a novel channel attention fusion module is developed, which can blend the deep representations at the channel dimension dynamically. Extensive experiments show that SC-HVPPNet notably boosts video restoration quality, with average bitrate savings of 5.29%, 12.42%, and 13.09% for Y, U, and V components in the VTM-11.0-NNVC RA configuration.