Advancing Learned Video Compression with In-loop Frame Prediction

TL;DR

This paper introduces ALVC, a novel learned video compression method with in-loop frame prediction that enhances compression efficiency by predicting frames from prior ones without additional bit-rate, achieving state-of-the-art results.

Contribution

The paper proposes an in-loop frame prediction module integrated into end-to-end learned video compression, improving reference quality and compression performance.

Findings

Outperforms existing learned video compression methods in PSNR and MS-SSIM.

Surpasses default hierarchical B mode of x265 in quality metrics.

Achieves state-of-the-art results in learned video compression.

Abstract

Recent years have witnessed an increasing interest in end-to-end learned video compression. Most previous works explore temporal redundancy by detecting and compressing a motion map to warp the reference frame towards the target frame. Yet, it failed to adequately take advantage of the historical priors in the sequential reference frames. In this paper, we propose an Advanced Learned Video Compression (ALVC) approach with the in-loop frame prediction module, which is able to effectively predict the target frame from the previously compressed frames, without consuming any bit-rate. The predicted frame can serve as a better reference than the previously compressed frame, and therefore it benefits the compression performance. The proposed in-loop prediction module is a part of the end-to-end video compression and is jointly optimized in the whole framework. We propose the recurrent and the bi-directional in-loop prediction modules for compressing P-frames and B-frames, respectively. The experiments show the state-of-the-art performance of our ALVC approach in learned video compression. We also outperform the default hierarchical B mode of x265 in terms of PSNR and beat the slowest mode of the SSIM-tuned x265 on MS-SSIM. The project page: https://github.com/RenYang-home/ALVC.