HyTIP: Hybrid Temporal Information Propagation for Masked Conditional Residual Video Coding

TL;DR

HyTIP introduces a hybrid video coding framework combining output-recurrence and hidden-to-hidden mechanisms, achieving superior rate-distortion performance with smaller buffer sizes compared to existing methods.

Contribution

This work proposes HyTIP, a novel hybrid buffering strategy that integrates explicit decoded frames and latent features for improved video coding efficiency.

Findings

Outperforms individual recurrence approaches in coding performance.

Achieves comparable results to state-of-the-art with smaller buffers.

Outperforms VTM 17.0 in PSNR-RGB and MS-SSIM-RGB metrics.

Abstract

Most frame-based learned video codecs can be interpreted as recurrent neural networks (RNNs) propagating reference information along the temporal dimension. This work revisits the limitations of the current approaches from an RNN perspective. The output-recurrence methods, which propagate decoded frames, are intuitive but impose dual constraints on the output decoded frames, leading to suboptimal rate-distortion performance. In contrast, the hidden-to-hidden connection approaches, which propagate latent features within the RNN, offer greater flexibility but require large buffer sizes. To address these issues, we propose HyTIP, a learned video coding framework that combines both mechanisms. Our hybrid buffering strategy uses explicit decoded frames and a small number of implicit latent features to achieve competitive coding performance. Experimental results show that our HyTIP outperforms the sole use of either output-recurrence or hidden-to-hidden approaches. Furthermore, it achieves comparable performance to state-of-the-art methods but with a much smaller buffer size, and outperforms VTM 17.0 (Low-delay B) in terms of PSNR-RGB and MS-SSIM-RGB. The source code of HyTIP is available at https://github.com/NYCU-MAPL/HyTIP.