ResiComp: Loss-Resilient Image Compression via Dual-Functional Masked Visual Token Modeling

TL;DR

ResiComp introduces a neural image compression framework that enhances error resilience against packet loss by integrating feature-domain concealment and masked visual token modeling, balancing compression efficiency and robustness.

Contribution

The paper presents a novel neural image codec with unified entropy modeling and packet loss concealment inspired by large language models, improving error resilience in NICs.

Findings

Significantly improves NIC robustness to packet loss.

Offers flexible modes to balance efficiency and resilience.

Demonstrates superior performance in experiments.

Abstract

Recent advancements in neural image codecs (NICs) are of significant compression performance, but limited attention has been paid to their error resilience. These resulting NICs tend to be sensitive to packet losses, which are prevalent in real-time communications. In this paper, we investigate how to elevate the resilience ability of NICs to combat packet losses. We propose ResiComp, a pioneering neural image compression framework with feature-domain packet loss concealment (PLC). Motivated by the inherent consistency between generation and compression, we advocate merging the tasks of entropy modeling and PLC into a unified framework focused on latent space context modeling. To this end, we take inspiration from the impressive generative capabilities of large language models (LLMs), particularly the recent advances of masked visual token modeling (MVTM). During training, we integrate MVTM to mirror the effects of packet loss, enabling a dual-functional Transformer to restore the masked latents by predicting their missing values and conditional probability mass functions. Our ResiComp jointly optimizes compression efficiency and loss resilience. Moreover, ResiComp provides flexible coding modes, allowing for explicitly adjusting the efficiency-resilience trade-off in response to varying Internet or wireless network conditions. Extensive experiments demonstrate that ResiComp can significantly enhance the NIC's resilience against packet losses, while exhibits a worthy trade-off between compression efficiency and packet loss resilience.