Viewport-based Neural 360{\deg} Image Compression

TL;DR

This paper introduces a viewport-based neural compression method for 360-degree images that reduces bit usage by 14% without quality loss, using a transformer-enhanced global information sharing approach.

Contribution

It proposes a novel viewport extraction and compression pipeline with a transformer-based ViewPort ConText module to better preserve global information in 360-degree image compression.

Findings

Achieves 14% bit savings over existing methods.

Outperforms traditional 2D codecs in viewport-based compression.

Effectively captures global prior information across viewports.

Abstract

Given the popularity of 360{\deg} images on social media platforms, 360{\deg} image compression becomes a critical technology for media storage and transmission. Conventional 360{\deg} image compression pipeline projects the spherical image into a single 2D plane, leading to issues of oversampling and distortion. In this paper, we propose a novel viewport-based neural compression pipeline for 360{\deg} images. By replacing the image projection in conventional 360{\deg} image compression pipelines with viewport extraction and efficiently compressing multiple viewports, the proposed pipeline minimizes the inherent oversampling and distortion issues. However, viewport extraction impedes information sharing between multiple viewports during compression, causing the loss of global information about the spherical image. To tackle this global information loss, we design a neural viewport codec to capture global prior information across multiple viewports and maximally compress the viewport data. The viewport codec is empowered by a transformer-based ViewPort ConText (VPCT) module that can be integrated with canonical learning-based 2D image compression structures. We compare the proposed pipeline with existing 360{\deg} image compression models and conventional 360{\deg} image compression pipelines building on learning-based 2D image codecs and standard hand-crafted codecs. Results show that our pipeline saves an average of $14.01\%$ bit consumption compared to the best-performing 360{\deg} image compression methods without compromising quality. The proposed VPCT-based codec also outperforms existing 2D image codecs in the viewport-based neural compression pipeline. Our code can be found at: https://github.com/Jingwei-Liao/VPCT.